Add *Locks as a separate modifiers

[eugenesvk]

• Added support for using CapsLock,NumLock, ScrollLock, Fn, FnLock, KanaLock, Loya, Roya, Symbol, SymbolLock as separate modifiers.

• Implemented state update on Windows for CapsLock,NumLock, ScrollLock, KanaLock, Loya, Roya. Don't know how to add the others as they don't even have VKs

(while l/roya seem idential to L/Rshift, there is no Oya state unlike with Shift, so each is added separately to the supposedly side-agnostic "state") src

• Windows Tested on all platforms changed
• Added an entry to the changelog module if knowledge of this change could be valuable to users
• Updated documentation to reflect any user-facing changes, including notes of platform-specific behavior
• Created or updated an example program if it would help users understand this functionality

On top of previous modifiers changes #4237 to have fewer conflicts and use existing example app

Partially addresses #1426

[kchibisov]

Are there any other modifiers from the spec that are missing? If so, we should just add all of them at this point.

Before we move further with those keyboard related PRs, @madsmtm what's the status of keyboard-types stuff? I think what we're doing now is pretty much is bringing stuff in line with how modifiers are in w3c spec, which is fine by me.

Though, I guess those won't conflict.

[eugenesvk]

Oh, I'd like to add everything and more (since the spec is limited), but I didn't since I had no way for immediately testing it.

https://w3c.github.io/uievents-key/#keys-modifier

It's missing 4 keys from the spec:

• 2 fn/fnlock - but I don't have a way to test those since they don't get to the OS
• 2 sym/symlock - never seen this one

(and it seems those don't even have any VKs on Windows, so can't test anything here, you'd have to us scan codes for these keys if they existed, but that would be hardware-dependent?)

Do you know of any Windows/Mac virtual keyboard app that can simulate those to check how Winit treats them?

It's also missing 3 niche useful keys not in the spec (and these exist on some keyboards I think unlike the symbols key which per spec is on virtual keys):

• 1 toggle modifier Kana (like Caps) VK_KANA on Win
• 2 regular modifiers Loya, Roya (non-toggle like Shift) VK_OEM_FJ_ROYA and VK_OEM_FJ_LOYA

If that's ok to just add those for completeness without platform impls, could do that.

[eugenesvk]

By the way, there is one big issue with keyboard-types - its modifiers repeat the common mistake (also encoded in Web spec) of having no side-awareness. Winit partially has this issue one of its APIs, but it has an escape hatch of having side-aware modifiers via a separate API.

My strong suggestion is instead of doubling down on this limited design to instead build a proper single side-aware foundation (and also add all the keys that aren't in the web spec)

[kchibisov]

It's missing 4 keys from the spec:

2 fn/fnlock - but I don't have a way to test those since they don't get to the OS
2 sym/symlock - never seen this one

Whether those will have impl or not is a different matter, since we discuss the top-level API for those here. So I don't see a point in not adding those just a variants even though, they are never built by backends provided by winit for now.

macOS does have fn modifier exposed though. sym/symlock I have no idea what this is.

By the way, there is one big issue with keyboard-types - its modifiers repeat the common mistake (also encoded in Web spec) of having no side-awareness. Winit partially has this issue one of its APIs, but it has an escape hatch of having side-aware modifiers via a separate API.

No one really stops to put it along the way, since modifiers are generally not side awere and e.g. on some backends you have no way to know that information at all, so the way web spec does it is reasonable. Some modifiers are not side aware as well, like CapsLock. I really don't mind having side awareness in a separate struct like it's now. Also for bindings you basically never need side awareness (I've never seen in my entire life a side aware binding, some apps have just modifier press, but that is done just by KeyEvent handling like any other button, not by checking the modifier state).

[eugenesvk]

Ok, will add all those extra without test impls

since modifiers are generally not side awere

Well, fundamentally they always are since they're always driven by real/virtual keys that can be referenced with side-awareness, it's just that you're right, sometimes that information is lost when it comes to Winit, but...

on some backends you have no way to know that information at all

This is not a blocker

In cases when you don't get side-aware information from the OS you could always store side-unaware state in the Left bitflag (even today side-aware state isn't reliable, so this won't change).

#4236 (comment)

so the way web spec does it is reasonable

It's not reasonable to destroy valuable info. This makes shortcuts wasteful. For example, you have two best key locations - thumb keys - 50% wasted since they're by default treated as a single side-agnostic mod Alt (Win) Cmd(Mac).

Some modifiers are not side aware as well, like CapsLock.

That's fine, I'm not suggesting we make up imaginary sides where none ever existed, so it will be a single CapsLock bitflag (and only 2 paired L/RShifts)

I really don't mind having side awareness in a separate struct like it's now.

But that's a waste, this is artificial separation that makes the APIs more complicated/confusing

Also for bindings you basically never need side awareness

You're almost using one yourself :)! From another issue you mentioned that your right alt is a composable symbols key, that's almost side awareness! Only in your case you need to have a special OS feature / app. But in a better world you could have that encoded in your RAlt+X keyboard layout layer and than your RAlt+X would insert a symbol from your keyboard layout instead of activating Alt+X shortcut.

I've never seen in my entire life a side aware binding

I'm using a few myself, but fundamentally this is a chicken&egg issue. Since almost everywhere the defaults are side-agnostically bad and either hard/impossible to fix, you don't see many uses. That's the benefit of having a simpler API so that it's not harder to do the right thing

[kchibisov]

You're almost using one yourself :)! From another issue you mentioned that your right alt is a composable symbols key, that's almost side awareness! Only in your case you need to have a special OS feature / app. But in a better world you could have that encoded in your RAlt+X keyboard layout layer and than your RAlt+X would insert a symbol from your keyboard layout instead of activating Alt+X shortcut.

it's IME/compose it's a bit different beast, since you have input that you can cancel, I'm not typing like 1 char with those, more like those: ¯\_(ツ)_/¯.

I'm using a few myself, but fundamentally this is a chicken&egg issue. Since almost everywhere the defaults are side-agnostically bad and either hard/impossible to fix, you don't see many uses. That's the benefit of having a simpler API so that it's not harder to do the right thing

The point is that when this is a common modifier API that may suggest that those are generally should be present. However side aware information is generally very limited, like on Wayland I have no idea how you'd get it for example in a reliable way, like my keyboard for example has 2 left shifts(for reasons, but I still have shift + shift binding possible, because the way how modifiers are applied). Like having a separate location for extra info, because this information is generally limited to only some OS and shouldn't be relied on, sounds sane to me.

[eugenesvk]

Yeah, I know, we've discussed this difference between your compose and AltGr-like -modifier input, IMEs are built "on top" of regular layouts, I was just traslating how that extra layer could behave at the base with side-awareness.

[kchibisov]

Yeah, I know, we've discussed this difference between your compose and AltGr-like -modifier input, IMEs are built "on top" of regular layouts, I was just traslating how that extra layer could behave at the base with side-awareness.

Sure, but we have a rather bad example of that is macOS if you want that sort of side awareness, where you don't have AltGr and app have to make a side aware handling of Alt vs Right Alt itself, it's just really messy and dedicated modifier is way better, since your unicode input shouldn't have any modifier attach to it rather than maybe Shift, since otherwise it won't make much sense and would be a disaster to handle in the bindings.

[eugenesvk]

app have to make a side aware handling of Alt vs Right Alt itself, it's just really messy

Maybe Winit could have a config to treat right alt as AltGr?
But also, messy compared to what?

and dedicated modifier is way better

Sure, and that's the reason why on Windows some people change RAlt to ROya. But afaik Mac layout simply doesn't support it! So messy beats impossible...

[kchibisov]

Maybe Winit could have a config to treat right alt as AltGr?
But also, messy compared to what?

We have that just for macOS, yes. Messy compared to anything else.

[eugenesvk]

The point is that when this is a common modifier API that may suggest that those are generally should be present.

This is currently not true with Winit, lshift_state/rshift_state IS a common modifier API, it's right there in the core https://docs.rs/winit/latest/winit/event/struct.Modifiers.html and is MORE accessible than the side-agnostic state which requires separate .state() call

However side aware information is generally very limited,

Well, how limited is it? On Windows it's present (including sticky) except for both keys pressed. On a Mac it's mostly present except for sticky mods.

like on Wayland I have no idea how you'd get it for example in a reliable way

Ok, so there you would continue to only use the side-agnostic API

my keyboard for example has 2 left shifts

But what's the logical layout like? Does your layout have 2 VK_LShifts (whatever is vk equivalent on Wayland)? If so, then from the logical modifier perspective these are identical, so this is not an issue for a side-aware modifiers API.

Like having a separate location for extra info, because this information is generally limited to only some OS and shouldn't be relied on, sounds sane to me.

But this doesn't change, you can continue to have the same side-agnostic .shift() use, without an extra call. The reliability guarantees are not affected by this change

[eugenesvk]

We have that just for macOS, yes.

Is this the OptionAsAlt config (a bit of a misnomer since on some physical mac keyboars option IS labeled as Alt)? Maybe could use the new AltGr and become OptionAsAltGraph :)

Messy compared to anything else.

But what is "anything else"? The different modifier is not supported in the layout, andif your keyboard can't replace RAlt with a different mod, and your app doesn't want to care about Alt sides, what else is there?

[kchibisov]

Well, how limited is it? On Windows it's present (including sticky) except for both keys pressed. On a Mac it's mostly present except for sticky mods.

On Wayland/X11 you don't have modifier event when you already have e.g. Shift pressed and you press it one more time, you can try to check the key manually, but modifiers on Wayland/X11 are decoupled from the key itself generally, so the information will be missed half of the time, since you simply don't have that data once you e.g. don't actively type into the window/etc/etc.

Is this the OptionAsAlt config (a bit of a misnomer since on some physical mac keyboars option IS labeled as Alt)? Maybe could use the new AltGr and become OptionAsAltGraph :)

It's AltGr by default on macOS, yes, both alts are AltGr on macOS.

But what is "anything else"? The different modifier is not supported in the layout, andif your keyboard can't replace RAlt with a different mod, and your app doesn't want to care about Alt sides, what else is there?

The point is that macOS is AltGr by default on both sides, to have a regular one you have to go through some things. macOS is also weird since they have Fn as a regular modifier now and bindings with Fn, which are handled on a client side, but that's irrelevant for this issue...

[eugenesvk]

Added all 7 extra mods and implemented in Windows whatever had VKs and could be mapped in a layout to test

[eugenesvk]

On Wayland/X11 you don't have modifier event when you already have e.g. Shift pressed and you press it one more time

That's the same issue with Windows "except for both keys pressed".

you can try to check the key manually, but modifiers on Wayland/X11 are decoupled from the key itself generally, so the information will be missed half of the time, since you simply don't have that data once you e.g. don't actively type into the window/etc/etc.

This seems to be different since on Mac/Win you can (in common cases) reliably check for the state in a side-aware manner (on Win by checking mod key state, on Mac by checking event flags)

The point is that macOS is AltGr by default on both sides

I understand that Option=Alt acts like AltGr, but API-wise it's still set as Alt modifierstate/modifierkey, right? I mean, there is currently simply no AltGr modifier in the types. And I've added only one without side separation.

But if you realy meant that mac's Alts by default should be treated as AltGrs unless you set some special config (default OptionAsAlt is None), then there should also be 2 AltGrs, left and right, to express that.

Maybe this illustration would help: let's say in your app you've defined Alt+Q to quit (both on Windows and macOS; either left or right mod).
What do you expect to happen with Windows layout having physical Right Alt as AltGr and left alt as usual, and macOS having the default layout?

• Window should quit on physical LeftAlt+Q since it will set Alt modifier, which will match
• Window should NOT quit on physical RightAlt+Q since (with the updated modifiers) a separate AltGraph modifier would be set, so Alt+Q won't match
• but then should physical left/right Alt+Q on a Mac NOT quit because by default these Alts behaves like AltGr, so should also be APIed as AltGr as well and insert special symbols?

macOS is also weird since they have Fn as a regular modifier

That's a great Mac part! On Win laptops that's usually a wasted key