Support WM_GETOBJECT on Windows

[ajtribick]

It would be useful to allow Windows applications to respond to WM_GETOBJECT messages in order to support building accessible user interfaces via Windows UI automation. As far as I can see, winit does not currently allow access to this event. Since accessibility both widens the potential user base and may be a legal requirement, it would be good to support this.

[maroider]

I don't know if Winit should deal with accessibility directly, but linebender/druid#299 has some interesting links on the topic.

You can actually just subclass the window and handle WM_GETOBJECT yourself, although this isn't necessarily the most elegant of solutions.

[ajtribick]

Thanks for the info, that's pretty useful!

I'm also not sure that accessibility is necessarily in scope for Winit. On the other hand, Winit does own the message loop and on Windows part of the accessibility API involves a specific message being passed into that loop. Having some minimal way to supply a handler would be useful in the Windows-specific API, although I agree that going much beyond that would be out-of-scope. As I understand it, Linux and Mac don't operate this way (no idea about mobile/web) and can be handled without needing to handle specific event types. Providing a cross-platform API over accessibility would be best left to another crate.

Nevertheless I will have a look at trying to do this via subclassing.

[mwcampbell]

One option would be to directly integrate support for my AccessKit project, which I just published to crates.io for the first time. I implemented a direct integration of AccessKit into winit on this branch. I understand if the winit developers don't want this kind of tight coupling. I will probably add a function in the AccessKit Windows adapter to do the window subclassing hack, but having that be the only solution just seems so ugly. I'm open to other ideas.

Note that while AccessKit aims to be cross-platform, only the Windows implementation is ready so far.

[maroider]

I agree that subclassing the window isn't pretty, but direct integration like what you've implemented is something I consider to be unlikely to be added to winit proper. I've been thinking a bit about how to let users handle events which winit doesn't handle, but that's not something I've "put out there" yet.

PS: Congratulations on publishing AccessKit to crates.io.

[mwcampbell]

OK, then I think what I'll do is create an accesskit_winit crate that takes a newly created winit window and does the necessary platform-specific steps, e.g. Win32 window subclassing, to hook up AccessKit. The only complication there is that in order for assistive technologies (e.g. screen readers) to observe a consistent state, the caller will have to create the window invisible, set up the AccessKit integration, then show the window. To make sure that happens, I could have accesskit_winit panic if the window is already visible.

[maroider]

The only complication there is that in order for assistive technologies (e.g. screen readers) to observe a consistent state, the caller will have to create the window invisible, set up the AccessKit integration, then show the window. To make sure that happens, I could have accesskit_winit panic if the window is already visible.

Well, that's unfortunate. I'll see if I can think of some way of changing winit's API to better accommodate this.

[ndarilek]

I certainly don't want to make it seem like I'm piling on here, but I hope we can figure out a way to bake this into winit proper. I'm afraid that if we don't make the right thing automatic, that we'll be patching a bunch of game engines/UI toolkits to pull in an extra accesskit_winit crate and change their startup.

Thanks!

[mwcampbell]

I have a new proposal for giving winit hooks to cleanly support accessibility providers such as AccessKit, without directly coupling winit to AccessKit itself. To give credit where it's due, this idea was sparked by a discussion yesterday on the #plugin-gui channel of the Rust Audio discord, when I proposed directly integrating AccessKit into the baseview crate.

I propose a crate called something like raw-a11y-provider, as a neutral interface analogous to the current raw-window-handle crate. Perhaps this new crate could eventually be owned by the Rust Windowing organization. It would have a platform-specific trait with the minimal set of methods to handle top-level accessibility requests on each platform. I could define each platform's version of the trait in a platform-specific module, then import the correct one in lib.rs.

The Windows version of the trait would look like this:

pub trait RawAccessibilityProvider {
    fn wm_getobject(&self, wparam: WPARAM, lparam: LPARAM) -> LRESULT;
}

The macOS version would look like this:

pub trait RawAccessibilityProvider {
    fn view_children(&self) -> *mut NSArray<NSObject>;
    fn focus(&self) -> *mut NSObject;
    fn hit_test(&self, point: NSPoint) -> *mut NSObject;
}

These method signatures are mostly based on the public APIs of the AccessKit Windows and macOS adapters, but as far as I'm aware, they carry no assumptions related to the design of AccessKit in particular. (The Windows one is somewhat simplified from what's currently in AccessKit, reverting an inelegant solution to a problem I ran into last year. I believe I'll be updating that API in AccessKit soon.)

Where the platform-specific types come from is an open question. On Windows, the WPARAM, LPARAM, and LRESULT types are defined in the windows-sys crate, so I think the raw-a11y-provider crate should use windows-sys, to minimize the impact on build time. On macOS, AccessKit is using objc2, so we could pull the NSObject and NSArray types from there, but I'd be just as happy using raw pointers. We have to use an actual struct for the point parameter to hit_test, though.

On iOS and Android, the same principle applies, though the specific methods are different. On Unix and web, the trait would be empty. On Unix desktops using AT-SPI, as far as I'm aware, there's no real integration point with the windowing system; AT-SPI is off to the side, using D-Bus, and the application would simply instantiate an AT-SPI implementation on startup, if the AT-SPI bus is running. On the web platform, the application would generate a DOM tree parallel to the canvas.

So in AccessKit, the platform adapters would implement the platform-specific versions of this trait. On the winit side, I propose adding a new variant to WindowEvent, like this:

AccessibilityRequested {
    provider: &'a mut Option<Box<dyn RawAccessibilityProvider>>,
}

There is precedent for an event having an output field; see WindowEvent::ScaleFactorChanged. By making this an Option, we can make this minimal, low-overhead accessibility support an integral part of winit, not an optional feature. If the output is None after the event handler returns, then the platform implementation can do the appropriate default behavior, e.g. calling DefWindowProcW for the WM_GETOBJECT message on Windows, or calling the superclass definitions of the view methods on macOS.

The platform implementations would store the RawAccessibilityProvider trait object returned by the event handler in their window/view state. This way the provider only has to be initialized once, on the first request. That way the event handler doesn't have to be called with this event for every WM_GETOBJECT message or view method call. I'd suggest using OnceCell to hold the trait object. On Windows, the Box would need to be converted to an Rc, so the wm_getobject method can be called while not holding a lock (or runtime borrow) on the window state that contains the trait object. This is necessary because the WM_GETOBJECT handler in the provider (e.g. AccessKit) may call UiaReturnRawElementProvider, which may lead to reentrant handling of WM_GETOBJECT. I saw that happen while working on my original, direct integration of AccessKit in winit last year. I solved the problem at the time by having AccessKit's handle_wm_getobject return impl Into<LRESULT>. But I think having the platform implementation store the provider in an Rc is a better solution.

Ultimately, working code is worth a thousand words, so maybe I should just go ahead and create the raw-a11y-provider crate, then integrate it in a new fork of winit, to show how it would work. But I thought it might be worthwhile to get some feedback on this initial sketch of a design first.

Some readers may be aware that I have already implemented an accesskit_winit crate. It uses Win32 subclassing on Windows, and a roughly equivalent Objective-C runtime technique on macOS, to do the necessary overrides. In addition to being generally ugly, this approach has another, more concrete downside, which I have only recently come to appreciate. To support lazy initialization (only creating and maintaining the AccessKit tree if accessibility is actually requested), the adapter has to take a callback which returns an initial tree update. This callback takes no parameters, and it must be a closure with a static lifetime. This makes it cumbersome for the callback to access mutable state in the application or GUI toolkit. By contrast, by adding a new event to winit, we integrate cleanly into the callback system that winit already has, i.e. the event handler function. I believe this type of direct support in winit would enable me to fix a current wart in my work-in-progress integration of AccessKit in egui; for details, see my latest comment on my egui PR.

I look forward to any thoughts on this proposal.

cc @madsmtm, who previously expressed interest in implementing some kind of integration in winit to replace my current use of Objective-C subclassing.