Keyboard / Controller Support?

[velvitonator]

(ADMIN EDIT July 2016: see #323 (comment) for the current todo list / ongoing tasks)

I've been looking to make our ImGUI accept controller input, for use on consoles. Something like hitting d-pad down to select the next item down, etc.

From what I could tell, however, the only way to do such a thing would be to spoof mouse control. Is this the case? If so, is there a way to ask for the rectangle of a given item outside of the update loop?

I also couldn't find any way to programmatically set the focus of an item other than input text fields; did I miss something?

[ocornut]

Before anything else, have you looked into using a system like Synergy + usynergy.c to have your game console use your PC mouse/keyboard ? I'd wholly recommend that.

If so, is there a way to ask for the rectangle of a given item outside of the update loop?

Not sure what you mean by "the update loop" nor why you would need that if you are spoofing mouse control? If you are going to spoof mouse control just make your controller adjust the MousePos and MouseDown (buttons) values.

You can call GetItemRectMin() / GetItemRectMax() to obtain the bounding rectangle of the last item (which might be clipped and outside of view, testable with IsItemVisible()) but I don't see how that would help here.

I also couldn't find any way to programmatically set the focus of an item other than input text fields; did I miss something?

No it's not available yet, there's no generalized concept of focus in fact.

Now if you really want controller support. There's different way we can approach this:

• Emulating mouse with game controller (or keyboard) would be easy to implement and functional immediately. Obviously it'd be rather awkward and underwhelming interaction-wise, but for casual use it may be enough.
• If you only want this for simple sort of UI like list of items you can emulate the concept of "whats focused" on user side and handle interaction yourself (e.g. Button X pressed while Item N is selected).
• The correct thing would be to fully handle keyboard/controller navigation and per-widget focus. It's not a simple feature, probably a bunch of work to get it right (I'd say 1 week optimistically). It is a very desirable feature but I don't know when / if I'll ever be able to do it.

I have rename your topic to keyboard/controller because the problems are the same for keyboard controls and many people are likely to come from the angle of desiring keyboard control.

[velvitonator]

Before anything else, have you looked into using a system like Synergy + usynergy.c to have your game console use your PC mouse/keyboard ? I'd wholly recommend that.

Yeah, we've got this up and running (except for keyboard support, since I've yet to hook up the manual character-input stuff). We want to add controller as a good native option, so users don't have to switch input devices.

Not sure what you mean by "the update loop" nor why you would need that if you are spoofing mouse control? If you are going to spoof mouse control just make your controller adjust the MousePos and MouseDown (buttons) values.

Apologies; I meant that we could do something like move the mouse down "an item" instead, but that starts to get a bit complicated for client code, especially since we'd need this to be something for everyone to use, to make their own debug GUIs. With that approach, we'd need the screen-rect of the options etc.
This is all certainly possible for us to do, it's just a matter of the effort we want to expend on it. Hence, I'm hoping to clarify the available options before embarking on something time-consuming. =)

Emulating mouse with game controller (or keyboard) would be easy to implement and functional immediately. Obviously it'd be rather awkward and underwhelming interaction-wise, but for casual use it may be enough.

Yeah, this is looking to be the only option in the timeframe we're looking at. Since we also have Synergy, I'm not convinced adding this is worth it; I know I'd easily switch to mouse + keys as soon as I had to, rather than bear analog-stick mouse cursor!

If you only want this for simple sort of UI like list of items you can emulate the concept of "whats focused" on user side and handle interaction yourself (e.g. Button X pressed while Item N is selected).

Certainly! But getting it to reflect in the UI is the tricky bit =/
Also, we already have a couple of in-game debug UIs using ImGUI, and they have quite a bit of variety. =P

The correct thing would be to fully handle keyboard/controller navigation and per-widget focus. It's not a simple feature, probably a bunch of work to get it right (I'd say 1 week optimistically). It is a very desirable feature but I don't know when / if I'll ever be able to do it.

Hmm, I'll be honest: I'm so accustomed to working with closed-source things that simply adding the feature to ImGUI didn't occur to me. I think it's a little out of scope for the timeframe I have, but it's certainly worth considering in the longer term, should we find that controller support is indispensable for whatever reason.

I have rename your topic to keyboard/controller because the problems are the same for keyboard controls and many people are likely to come from the angle of desiring keyboard control.

Good idea!

[ocornut]

Apologies; I meant that we could do something like move the mouse down "an item" instead, but that starts to get a bit complicated for client code, especially since we'd need this to be something for everyone to use,

It's trivial to store the rectangle of the hovered item (we can add that to the library), so you could move the cursor outside of it but you wouldn't know by how much as you don't know where the next item. An hybrid approach to move outward the item by the amount of its width/height and then move smoothly if there's no item under the cursor. It would be easy to implement and may be a little better than a cursor but probably still mediocre. If you are on PS4, I haven't tried it but I suspect the touch pad is precise enough to emulate a mouse.

Hmm, I'll be honest: I'm so accustomed to working with closed-source things that simply adding the feature to ImGUI didn't occur to me. I think it's a little out of scope for the timeframe I have, but it's certainly worth considering in the longer term, should we find that controller support is indispensable for whatever reason.

FYI if your project/company has the fund I'm more than happy to provide custom work on imgui to develop that sort of non-trivial feature (been looking for possible ways to fund its development as I can't really afford it any more myself, at least not at the pace I was working on it a few months ago). Something to consider, may be more optimal than you doing it and it'll benefit the sustainability of the library.

It's one of the those feature that's touching quite a lot of elements in the code - quite a complex feature to do right. If you to want to tackle it yourself I can also guide you.

[velvitonator]

Sorry for the silence; I forgot the re-mark the mail as unread to get back to it.

Your idea to use the PS4 touchpad was pretty effective--it's somewhat finicky, but much better than a stick, and it wasn't too difficult to hook up the sticks and pad for scrolling, either.

So, we're good for now; thanks for the suggestion!

[ocornut]

I don't have an ETA yet but I've been working on this lately.
When it starts to be semi usable I will push it to a public branch, right now it is way too prototypey.

[ocornut]

Some progress report, been improving support for the popup stack, combos, menus, sliders, visualizing selection, avoiding confusion between keyboard and mouse, etc.
There's still a billion things to do, it is a little overwhelming.

[Pagghiu]

This is VERY interesting 😁

[MrMarkie]

This looks great. We'd like to use it for interaction with debug UI in PSVR. It's exactly the kind of thing we need for this. When are you looking to make it available?

[ocornut]

When it is done @MrMarkie :) I've been working on my non-existent spare time until now, but because this feature is now sponsored (a first) I'm aiming before end of July. Will probably start testing and iterating on API and edge cases mid-july.