Specify Multi-touch Support

[hsk81]

At www.dizmo.com, we're developing an interface (of things), and decided to go for VNC and noVNC to enable browser support for our software, which relies as well on multi-touch support. However, noVNC does not seem to have such a feature enabled.

Hence, would it be possible to have multi-touch integrated? We would be willing to also pay the developer who integrates this feature.

I've issued a bounty of 240 DAI (a stable-coin corresponding to 240 USD) to have this issue not solved, but fully specified. Please have a look at: https://gitcoin.co/issue/novnc/noVNC/944/1047

My goal with this bounty would be to have exhaustive specification, which would allow me to open a follow-up of issue where then the actual multi-touch support is implemented. To claim the bounty, please provide a pull request with a corresponding Markdown document containing the specification.

Please see also our video gallery at https://www.dizmo.com/video-gallery/, where we display some of our use cases and where we plan to use multi-touch. In our case noVNC is running within a Docker container, hence the specification should be compatible with such a context.

NOTE: The bounty is structured as a contest with an expiration date of two weeks.

[samhed]

This would mean an extension of the RFB protocol, it is not trivial work.

[hsk81]

Is it maybe possible to provide an estimate (in terms of developer hours) w.r.t. to this RFB protocol extension? I'm not familiar with RFB, hence the question: why would multi-touch imply such an extension?

[samhed]

If you mean that touch events should be passed to the desktop or application in the remote session, then we do need an extension to the protocol. The protocol only allows regular mouse or keyboard input at the moment.

No I can't give you an estimate. Estimates never work out well.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 240.0 DAI (240.0 USD @ $1.0/DAI) attached to it.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
• Want to chip in? Add your own contribution here.
• Questions? Checkout Gitcoin Help or the Gitcoin Slack
• $20,949.55 more funded OSS Work available on the Gitcoin Issue Explorer

[hsk81]

@samhed @CendioOssman @DirectXMan12 @kanaka I've issued a bounty of 240 DAI (a stable-coin corresponding to 240 USD) to have this issue not solved, but fully specified. Please have a look at:

• https://gitcoin.co/issue/novnc/noVNC/944/1047

My goal with this bounty would be to have exhaustive specification, which would allow me to open a follow-up of issue where then the actual multi-touch support is implemented. To claim the bounty, please provide a pull request with a corresponding Markdown document containing the specification.

Please see also our video gallery at https://www.dizmo.com/video-gallery/, where we display some of our use cases and where we plan to use multi-touch. In our case noVNC is running within a Docker container, hence the specification should be compatible with such a context.

NOTE: The bounty is structured as a contest with an expiration date of two weeks.

[oogetyboogety]

@hsk81
Which type of specification are you leaning towards?
RFB has been extended as a specification to support multitouch inputs, pressure sensitivity, and more via RFB GII standard extension in at least two major implementations that have code associated with them, starting here:
https://groups.google.com/forum/#!topic/tigervnc-devel/Zupu-o6JDNI
It appears in TurboVNC you have Remote Wacom Tablet mode which uses standard messages, as well as in TigerVNC.
https://github.com/TurboVNC/turbovnc/blob/2e0a0fbad308b3fb53695e472679cd9b5024d48b/unix/Xvnc/programs/Xserver/hw/vnc/init.c#L974

However, these implementations have limitations. A correct proposal in my mind would specify equations for ellipsis for each contact point and its coordinates. An application of this would then natively identify gestures, for example on a tablet you might expect two fingers to zoom but one handed phone you might expect pinch to zoom.
The same messages used in RDP could be added to the GII format as well:
https://msdn.microsoft.com/en-us/library/hh554669.aspx
Please let me know your feedback, thanks in advance

[hsk81]

@mul1sh yes, it is still open! Please note that I'm looking for a specification only (for now at least).

[hsk81]

@oogetyboogety The issue is that I'm myself not experienced enough w.r.t. multi-touch support in general. So your question about

Which type of specification are you leaning towards?

Is rather difficult to answer for me. What we have is the following software stack:

• our own cross-platform QT based application, which has multi-touch support;
• we have embedded it into a debian:jessie based docker container;
• the container runs x11-xserver, tigervnc and noVNC;

We don't have multi-touch support with noVNC, and the goal of the specification would be to clearly define a roadmap to have multi-touch implemented: This may or may not involve configuring or extending other software (like x11-xerver or tigervnc) besides noVNC.

But obviously we do not want to reinvent the wheel, and have the desired minimal functionality with as little effort as possible.

RFB has been extended as a specification to support multitouch inputs, pressure sensitivity, and more via RFB GII standard extension in at least two major implementations that have code associated with them, starting here: tigervnc-devel. It appears in TurboVNC#L974 you have Remote Wacom Tablet mode which uses standard messages, as well as in TigerVNC.

So, if we could bring in multi-touch support to noVNC using somehow the work already done for RFB (GII) and TigerVNC then great, let's do that: Here, I would like a specification (more like a roadmap), which would allow me to clearly judge if it is feasible and how long it would take to have multi-touch for noVNC.

However, these implementations have limitations. A correct proposal in my mind would specify equations for ellipsis for each contact point and its coordinates. An application of this would then natively identify gestures, for example on a tablet you might expect two fingers to zoom but one handed phone you might expect pinch to zoom. The same messages used in RDP could be added to the GII format as well: https://msdn.microsoft.com/en-us/library/hh554669.aspx

I'm not interested to create new specifications (or improve existing ones), if RFB has already such a specification in place. I would prefer an existing specification.

[frankchen07]

@oogetyboogety - are you still working on this issue? Just checking in :)