Touchscreen Mouse Events Not Captured

[lhstgithub]

I am finding that mouse events are triggered OK when using a physical mouse, however when using a touch screen in certain applications, Chrome and Edge so far tested, no mouse events are getting triggered.

Also the Mouse Move event does not trigger in Adobe Acrobat when scrolling a PDF, but click events do trigger OK.

This only happens when using a touch screen.

As soon as the mouse is outside of chrome or edge then the mouse events start triggering again.

I did find someone else is having the issue with low level mouse hooks, but not this code, so it maybe a general limitation

https://stackoverflow.com/questions/50407901/capturing-touch-events-with-lowlevelmouseproc-does-not-work-for-all-applications

Any help would be appreciated, thanks

[Visual-Vincent]

Apologies for my very late reply, I haven't really had the time lately to work on my personal projects.

As you found out yourself, you appear to not be the only one having this issue. I found a second Stack Overflow question also mentioning the same problem, where the OP was capturing mouse events (from a stylus rather than from touch, but still using low-level hooks) which works for most applications except browsers like Chrome, Edge, Internet Explorer, and VS Code (which uses Chrome to render it's UI): https://stackoverflow.com/questions/69364200/some-applications-absorb-mouse-events-made-by-stylus-so-that-i-cant-hook-them

Unfortunately I don't have a stylus or touch device, so I cannot test this behaviour myself, but I managed to find a post from one of the engineers working on Mozilla's Firefox browser on the XMBC forum (XMBC is an application used to remap your mouse buttons through low-level hooks), which may shed some light as to what the issue could potentially be: https://forums.highrez.co.uk/viewtopic.php?t=4632

According to the engineer, Firefox (and Chrome and Edge) render all the browser content (the websites you see) off-screen in a separate OS process. This is so that if the graphics renderer or rendering process crashes, it won't crash the entire web browser. They then create a separate "compositor window" which, if I understand things correctly, they embed inside the main browser window. This compositor window is responsible for taking the off-screen rendered browser content and displaying it on your screen. So essentially you have two separate windows for each browser window, which likely looks something like this:

I could be wrong, but this is based on my understanding of the Firefox engineer's description. This could however explain why the mouse hook, according to https://stackoverflow.com/q/69364200, works on the title bars of the web browsers, but not on the area which contain the website contents.

The engineer goes on to explain that after enabling a feature they call sandboxing for their render process in a Nightly build of Firefox, they started receiving reports from users that the aforementioned XMBC software was no longer working correctly with Firefox. The engineer explains that the sandboxing feature works by running the compositor window at Low integrity level. What this means is that Windows isolates processes in different integrity levels from each other, i.e. a process at a lower integrity level cannot modify a process at a higher integrity level (for more information see: Mandatory Integrity Control | Microsoft Docs).

The standard integrity level is Medium, which makes Firefox's main browser window run at Medium integrity level and the compositor window at Low integrity level. This means the compositor window cannot make changes to the main window, which is why XMBC stopped working after enabling this feature: It tried to post mouse messages to the compositor window, which in turn tried forwarding them to the main window, which it wasn't allowed to do.

The reason I mention this is because it could be relevant to your issue. It isn't clear to me what the exact cause of your issue is, but I do have some speculations:

• It could have something to do with that the compositor window of the browser runs at a lower integrity level, though how this would affect low-level hooks I don't know
• It could be that another application has installed a low-level hook (such as Chrome or Edge themselves) earlier in the chain which swallows the mouse events, preventing them from ever reaching your hook
• Or it could be a combination of both: An earlier hook (or driver?) in the chain prevents the mouse event directed towards the low-integrity window from reaching your hook. Maybe the touch device drivers does something that we are not aware of?

Unfortunately, as there is no way to debug or list all low-level hooks installed in the system, it's hard to diagnose the exact cause of the problem.

It would be interesting to see if you could try this with the Firefox browser as well and see if you experience the same issue.

Even though this was not a solution to the problem, I hope it helps shed some light on the situation.

[lhstgithub]

Thank you for such a detailed reply, it certainly does help, it will take me some time to look through it and investigate it all, but really appreciate the information.

Interestingly Firefox was the one browser that the touch events were detected with, so maybe they have the sandboxing feature turned off now with it causing issues for others as well.

[Visual-Vincent]

My pleasure! Always happy to see people getting use out of my library.

Just checked in my Firefox browser and sandboxing appears to be turned on. If you go into about:config and search for security.sandbox.gpu.level it should be set to 1.

If you can, it would also be helpful to see if you experience the same things as https://stackoverflow.com/q/69364200, where touch should work on Chrome's or Edge's title bar, tabs, etc., but not on the browser/website contents themselves. This would further support the assumption that something about the compositor window is the issue (whether it be another hook, integrity level, or something else).

[lhstgithub]

I do not have the touch screen with me at the moment as it was used for a project, but when I do I will test.

However when developing the app I did use Firefox and security.sandbox.gpu.level is set to 1 in it, I have not changed any settings in Firefox so this is how it will have been during testing and the mouse detection worked OK.

When I have the touch screen back soon I will test with Chrome and Edge to see if the title bars and tabs work as that would be helpful to narrow it down.