[Bug]: Screen flickering and voliatile image artifacts

[nitrotol]

Steps for reproducing the issue

1. Launch wluma

What is the buggy behavior?

Right after start app screen starts flickering (with frequency near 5 Hz) and sometimes appear dark rectangles on light parts of screen image which persist on the same place after change image (scroll out or change browser tab). I saw this only in Chrome but I can't be sure that it don't appear in other apps

What is the expected behavior?

Clear screen image without flickering

Logs

[2024-10-22T17:49:32Z DEBUG wluma] Using Config {
        als: Iio {
            path: "/sys/bus/iio/devices",
            thresholds: {
                0: "night",
                500: "bright",
                80: "dim",
                800: "outdoors",
                250: "normal",
                20: "dark",
            },
        },
        output: [
            Backlight(
                BacklightOutput {
                    name: "eDP-1",
                    path: "/sys/class/backlight/intel_backlight",
                    capturer: Wlroots,
                    min_brightness: 1,
                },
            ),
            Backlight(
                BacklightOutput {
                    name: "eDP-2",
                    path: "/sys/class/backlight/card1-eDP-2-backlight",
                    capturer: Wlroots,
                    min_brightness: 1,
                },
            ),
        ],
    }
[2024-10-22T17:49:32Z INFO  wluma] Continue adjusting brightness and wluma will learn your preference over time.
[2024-10-22T17:49:32Z DEBUG wluma::frame::capturer::wlroots] Using output 'Samsung Display Corp. 0x419D (eDP-1)' for config 'eDP-1'

Version

4.4.1

Environment

lsb_release -a; uname -a; pgrep -l sway; pacman -Q | egrep "(wlroots|vulkan|sway|clang|rust)"; dpkg -l | egrep "(wlroots|vulkan|sway|clang|rust)"; echo "WLR_DRM_NO_MODIFIERS=$WLR_DRM_NO_MODIFIERS"
zsh: command not found: lsb_release
Linux nitrotol-zenbook 6.11.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 10 Oct 2024 20:11:06 +0000 x86_64 GNU/Linux
1829 sway
1849 swaybg
1884 swayidle
egrep: warning: egrep is obsolescent; using grep -E
clang 18.1.8-3
grimshot-bin-sway 1-3
rust 1:1.81.0-1
sway-contrib 1.9.0-2
swaybg 1.2.1-1
swayfx 0.4-3
swayfx-debug 0.4-3
swayidle 1.8.0-2
swaylock 1.8.0-1
vulkan-headers 1:1.3.295-1
vulkan-icd-loader 1.3.295-1
vulkan-intel 1:24.2.4-1
vulkan-tools 1.3.269-1
vulkan-validation-layers 1.3.290-2
wlroots 0.18.1-1
wlroots0.17 0.17.4-3
zsh: command not found: dpkg
egrep: warning: egrep is obsolescent; using grep -E
WLR_DRM_NO_MODIFIERS=1

[maximbaz]

Wow, I've never heard anything like that 😧

Do you have two screens? Does it reproduce if you only leave one of them in the config?

[nitrotol]

Yeah, I have dual screen laptop

RUST_LOG=debug wluma
[2024-10-22T18:06:33Z DEBUG wluma] Using Config {
        als: Iio {
            path: "/sys/bus/iio/devices",
            thresholds: {
                20: "dark",
                0: "night",
                500: "bright",
                80: "dim",
                800: "outdoors",
                250: "normal",
            },
        },
        output: [
            Backlight(
                BacklightOutput {
                    name: "eDP-1",
                    path: "/sys/class/backlight/intel_backlight",
                    capturer: Wlroots,
                    min_brightness: 1,
                },
            ),
        ],
    }
[2024-10-22T18:06:33Z INFO  wluma] Continue adjusting brightness and wluma will learn your preference over time.
[2024-10-22T18:06:33Z DEBUG wluma::frame::capturer::wlroots] Using output 'Samsung Display Corp. 0x419D (eDP-1)' for config 'eDP-1'

Flickering still persist

[maximbaz]

I cannot possibly imagine how we can be affecting screen contents if all we do is receive a frame and copy it into a new image for further professing ☹️

What if you replace this entire part with let result = 0;, does flickering go away? This maps the frame but doesn't copy it.

wluma/src/frame/vulkan.rs

Lines 203 to 226 in 5e0732e

	self.begin_commands()?;
	let (target_mip_level, mip_width, mip_height) =
	self.generate_mipmaps(frame, &frame_image, &image);
	self.copy_mipmap(&image, target_mip_level, mip_width, mip_height);
	self.submit_commands()?;
	let pixels = mip_width as usize * mip_height as usize;
	let rgbas = unsafe {
	let buffer_pointer = self
	.device
	.map_memory(
	self.buffer_memory,
	0,
	vk::WHOLE_SIZE,
	vk::MemoryMapFlags::empty(),
	)
	.map_err(anyhow::Error::msg)?;
	std::slice::from_raw_parts(buffer_pointer as *mut u8, pixels * 4)
	};
	let result = compute_perceived_lightness_percent(rgbas, true, pixels);

Or what if you comment out this part, does the flickering go away? This is basically receiving frames and not doing anything with them at all.

wluma/src/frame/capturer/wlroots.rs

Lines 126 to 131 in 5e0732e

	let luma = self
	.vulkan
	.luma_percent(&frame)
	.expect("Unable to compute luma percent");
	controller.borrow_mut().adjust(luma);

[nitrotol]

Changes in vulkan.rs - flickering still persist
Changes in wlroots.rs - no changes, flickering still there

Maybe this is not about image as is, maybe it related to change backlight so fast?
But if it is, why I have black rectaggles here and there sometimes?

[maximbaz]

I did not expect that flickering would persist with changes to vulkan.rs!

Let's comment one more line and see if it changes anything, replace all of this please with let result = 0;, does it make a difference?

wluma/src/frame/vulkan.rs

Lines 201 to 226 in 77bac5a

	let (frame_image, frame_image_memory) = self.init_frame_image(frame)?;
	self.begin_commands()?;
	let (target_mip_level, mip_width, mip_height) =
	self.generate_mipmaps(frame, &frame_image, &image);
	self.copy_mipmap(&image, target_mip_level, mip_width, mip_height);
	self.submit_commands()?;
	let pixels = mip_width as usize * mip_height as usize;
	let rgbas = unsafe {
	let buffer_pointer = self
	.device
	.map_memory(
	self.buffer_memory,
	0,
	vk::WHOLE_SIZE,
	vk::MemoryMapFlags::empty(),
	)
	.map_err(anyhow::Error::msg)?;
	std::slice::from_raw_parts(buffer_pointer as *mut u8, pixels * 4)
	};
	let result = compute_perceived_lightness_percent(rgbas, true, pixels);

Similarly to test your idea that it is backlight change that could be the culprit, you could maybe try to replace this part with state.controller.as_mut().unwrap().adjust(0):

wluma/src/frame/capturer/wlr_export_dmabuf_unstable_v1.rs

Lines 155 to 162 in 77bac5a

	let luma = state
	.vulkan
	.as_ref()
	.unwrap()
	.luma_percent(state.pending_frame.as_ref().unwrap())
	.expect("Unable to compute luma percent");
	state.controller.as_mut().unwrap().adjust(luma);

By the way, I just pushed some major update to main branch, it should not have any difference whatsoever, but just so you know why references above have changed.

[nitrotol]

First case - error[E0425]: cannot find value frame_image in this scope
If keep line 201 - no changes, it still flicker

Please note that all previous tests I'm doing on tag 4.4.1
Now I checkout fresh main (just because tag 4.4.1 dont have wluma/src/frame/capturer/wlr_export_dmabuf_unstable_v1.rs

Fresh unchanged main - flickering
with changes in wlr_export_dmabuf_unstable_v1.rs - absolutely the same shit(

[maximbaz]

First case - error[E0425]: cannot find value frame_image in this scope
If keep line 201 - no changes, it still flicker

Right, sorry, comment out line 201 and all of the unsafe block in the end too, lines 229-234.

[nitrotol]

It's f**king magic
Somehow it stops flickering! On all versions (including installed 4.4.1). I don't understand how.
Maybe it depends on some gamma settings? I also have gammastep running...

Give me please some time to make more tests in different circumstances.
I'll keep you posted

[maximbaz]

A new wlr-screencopy-unstable-v1 protocol has landed in wluma's main branch, the way I understand how it is different is that we don't access the compositor's frame directly, but instead we create our own image and ask compositor to fill it in. So I'm guessing/hoping that this would have even less chance for wluma to cause any kind of image artifacts, if it always works on its own copy of the image.

Would you like to try it out? Just pull the latest main, keep capturer=wlroots, and if both protocols are supported, wluma will automatically pick the wlr-screencopy-unstable-v1 one (you will see in RUST_LOG=trace what it picks).

If you do get to try it, please report any issues you face with it, because I want to make this protocol a new default.

And also, on my own hardware I observed that with the new protocol I am getting occasional luma: 0, but I don't know if it's something that I do incorrectly in the code, or it's a bug in my driver.

$ RUST_LOG=trace cargo run
...
[2024-10-23T21:48:43Z TRACE wluma::predictor::controller] Prediction: 180 (lux: none, luma: 17)
[2024-10-23T21:48:43Z TRACE wluma::predictor::controller] Prediction: 181 (lux: none, luma: 16)
[2024-10-23T21:48:43Z TRACE wluma::predictor::controller] Prediction: 181 (lux: none, luma: 16)
[2024-10-23T21:48:43Z TRACE wluma::predictor::controller] Prediction: 181 (lux: none, luma: 16)
[2024-10-23T21:48:44Z TRACE wluma::predictor::controller] Prediction: 174 (lux: none, luma: 0)  <-------------
[2024-10-23T21:48:44Z TRACE wluma::predictor::controller] Prediction: 180 (lux: none, luma: 17)
[2024-10-23T21:48:44Z TRACE wluma::predictor::controller] Prediction: 181 (lux: none, luma: 16)
[2024-10-23T21:48:44Z TRACE wluma::predictor::controller] Prediction: 181 (lux: none, luma: 16)
[2024-10-23T21:48:44Z TRACE wluma::predictor::controller] Prediction: 181 (lux: none, luma: 16)
[2024-10-23T21:48:44Z TRACE wluma::predictor::controller] Prediction: 181 (lux: none, luma: 16)
[2024-10-23T21:48:44Z TRACE wluma::predictor::controller] Prediction: 181 (lux: none, luma: 16)
[2024-10-23T21:48:44Z TRACE wluma::predictor::controller] Prediction: 174 (lux: none, luma: 0)  <-------------
[2024-10-23T21:48:44Z TRACE wluma::predictor::controller] Prediction: 181 (lux: none, luma: 16)
[2024-10-23T21:48:45Z TRACE wluma::predictor::controller] Prediction: 174 (lux: none, luma: 0)  <-------------
[2024-10-23T21:48:45Z TRACE wluma::predictor::controller] Prediction: 181 (lux: none, luma: 16)
[2024-10-23T21:48:45Z TRACE wluma::predictor::controller] Prediction: 181 (lux: none, luma: 16)

Could you tell me if you see them? (they pop up relatively frequently).

If you don't see them, could you comment this line and try again?

wluma/src/frame/capturer/wayland.rs

Lines 431 to 432 in 8e81a8f

	// The frame is sadly not always ready at this stage, and I couldn't find how to make Vulkan synchronize with the compositor ☹️
	thread::sleep(Duration::from_millis(10));

I see them without sleep and don't see them anymore with sleep.

[nitrotol]

changes in src/frame/capturer/wlr_export_dmabuf_unstable_v1.rs don't helps at all, it still flickering
also with

comment out line 201 and all of the unsafe block in the end too, lines 229-234.

I tried fresh master with wlr-screencopy-unstable-v1 and I don't see luma: 0. Screen keeps flickering, and I see that each particular flick happend in the same time with each trace message appear. Same with commented 432 line in src/frame/capturer/wayland.rs

[maximbaz]

This might sound crazy but does your screen flicker when you screen share an entire screen (without wluma running) on https://mozilla.github.io/webrtc-landing/gum_test.html? I assume you use xdg-desktop-portal-wlr for that.

[maximbaz]

The reason I say this is because upon re-reading this thread I realized that I initially misread your message, specifically that after commenting both lines below still produces flickering:

 let luma = state 
     .vulkan 
     .as_ref() 
     .unwrap() 
     .luma_percent(state.pending_frame.as_ref().unwrap()) 
     .expect("Unable to compute luma percent"); 
  
 state.controller.as_mut().unwrap().adjust(luma); 

Basically, wluma consists of 3 parts: request screen contents from wlroots, import the image & get its "luma" value, predict & change brightness using luma value. The two code actions above correspond to the two latter activities, so by commenting them out, we are really left with just asking wlroots to get us screen contets, and nothing else - and any kind of screen sharing does that too!

[nitrotol]

In this case I havent flickering during fullscreen sharing

[maximbaz]

I'm completely at a loss, I'll be honest, I think the only way forward is to comment stuff out until we find what exactly we have to comment in order to stop flickering from happening.

I just pushed debug-flickering branch as a start, could you give it a go?

If it still flickers, could you also experiment with DELAY_SUCCESS value in wayland.rs? Increase it say to 1000, see if flickering is still there?

[nitrotol]

It's very strange...

• Clear debug-flickering - flickers
• DELAY_SUCCESS = 1000 - no flickering
• DELAY_SUCCESS = 2000 - no flickering
• DELAY_SUCCESS = 20 - no flickering

[nitrotol]

Also I find one more thing
I see the same flickering (localy in Chrome) when I opening a modal window with my media controls

It visible very clear on dark backgrounds and almost invisible with light ones

[maximbaz]

It sounds unpredictable, I can't explain why doing the same operation more and less frequently causes flickering. I suspect if you keep running this or main branch with any of those good DELAY_SUCCESS values, you will eventually get flickering too...

I think I would look for an issue elsewhere... Some other software, running in parallel? Maybe check BIOS for anything relevant to GPU? Just throwing some ideas... 🤔

[nitrotol]

As I mentioned before I have gammastep which also manipulated with a screen. And I have no options on BIOS related to GPU

Tomorrow I'll try to play with the "better" version and keep you posted

[maximbaz]

Cool thanks! I think although a lot of wluma users run gammastep too, I guess it wouldn't hurt to also see if you can confirm the flickering with only running wluma and nothing else - and if we for example know that flickering only happens when both apps run, we can't reliably say which one is to blame 😞

[maximbaz]

Found this project, which might be closer to wluma than just a screen sharing in a browser, maybe worth giving it a go and seeing if you see any artifacts using it: https://github.com/russelltg/wl-screenrec