Add "labwc over VNC" backend (Wayland)

[matt335672]

See #2637 (pinned issue) for the background on this.

This is more a show-and-tell than a PR at the moment. I though I'd put it out there to get some comments though. If we get somewhere useful with this I'll refactor a different PR for a merge.

First a couple of screenshots. This is a remmina client talking to an Xunutu25-04 backend. I've tested nothing else yet:-

Session is XFCE 4.20 running under the labwc compositor.

Working:-

1. chansrv drive redirection
2. chansrv clipboard (X11 only)

Not working:-

1. The obvious one is the screen resizing. I've not looked at this at all.
2. Audio plugins
3. chansrv clipboard (Wayland)

There are some code problems too:-

1. I've used a class hierarchy in sesexec to add labwc support next to X11 support. The code balance between the base and derived classes is not right at the moment.
2. We need a way to log output from labwc and wayvnc (maybe only for debugging)
3. I've got some file permission issues with the Wayland scripts on my test rig which need sorting out.
4. I'm doing some bad things with the environment variable list being passed into the session module.
5. The waitforx executable needs to be moved down into the sesexec directory, next to the wayland scripts.

The biggest issue to my mind is with X11 display numbers which are used extensively in xrdp. For X11 these are globals, and (effectively) always integers. Wayland displays are always strings. So the socket dir for a running session looks like this:-

$ sudo ls -l /run/xrdp/1001
total 0
srw-rw---- 1 testuser root 0 Aug 13 11:38 xrdpapi_wayland-0
srw-rw---- 1 testuser root 0 Aug 13 11:39 xrdp_chansrv_audio_in_socket_wayland-0
srw-rw---- 1 testuser root 0 Aug 13 11:39 xrdp_chansrv_audio_out_socket_wayland-0
srwxrwxr-x 1 testuser root 0 Aug 13 11:38 xrdp_display_wayland-0

I've tried to keep compatibility with number for X11, but this is distorting the code in places. I think it might be better to use strings like x11-10 (e.g. for X11 display 10) rather than a straight 10. This will need a change to the audio drivers, but we can stick with the plain number for xorgxrdp support.

I'm quite busy IRL for a week or so. After that, I'll start work on fixing deficiencies and trying other platforms, in particular Trixie which is now out.

Comments/questions are very welcome!

[lcniel]

Wow, this is quite a monumental work! The refactoring aspects look good as well. I really appreciate the work you put into this project and the way you interact with the user community of xrdp.

I will give this a build and a test run as soon as I have time.

[lcniel]

Can confirm that it works fine on my Rocky 10 build, no apparent issues thus far, other than the mentioned screen resizing one. The only thing I had to do besides turning off drdyvnc was to create an .xsession file in my user's homedir to run xfce4-session as I do with any other xrdp version.

Again, awesome job!

[matt335672]

Thanks for that @lcniel

I've not gotten to the bottom of the resize yet. The resize state machine triggers, but then labwc resizes back to a smaller size. I've added some (untested) code to enable logging for labwc and wayvnc which might help to debug this.

[lcniel]

Thanks for that @lcniel

I've not gotten to the bottom of the resize yet. The resize state machine triggers, but then labwc resizes back to a smaller size. I've added some (untested) code to enable logging for labwc and wayvnc which might help to debug this.

It's weird. I had some occasional wonkiness with interactive resizing with TurboVNC when connected to WayVNC (like the configuration resetting once connected, I had to change it away from server-side resizing every time, but that might just be some RFB implementation thing), but most of the time it worked without issue.

[matt335672]

I'm fairly confident the resize issue is an xrdp problem.

Resizing is a complicated dance involving the client and server. wayvnc is using an additional message in its ExtendedDesktopSize response which essentially informs xrdp the resize request has been forwarded. We don't cope well with that message. I suspect what's happening is the resize to the client screen size is happening, and then it is getting overridden when the server reports its original screen size late. I'm not completely sure about this, but I'm sure it's a fixable problem. I'm a little busy IRL for the next week or so, but I'll get to it when I can.

Debug logging for both labwc and wayvnc can now be enabled in sesman.ini.

[lcniel]

I'm fairly confident the resize issue is an xrdp problem.

Resizing is a complicated dance involving the client and server. wayvnc is using an additional message in its ExtendedDesktopSize response which essentially informs xrdp the resize request has been forwarded. We don't cope well with that message. I suspect what's happening is the resize to the client screen size is happening, and then it is getting overridden when the server reports its original screen size late. I'm not completely sure about this, but I'm sure it's a fixable problem. I'm a little busy IRL for the next week or so, but I'll get to it when I can.

Debug logging for both labwc and wayvnc can now be enabled in sesman.ini.

I had a look at the neatvnc source code: https://github.com/any1/neatvnc/blob/4962e0af5d550b2c4fd34c79dbdca543f233d87c/src/server.c#L1632

and that looks like exactly what happens, yeah (and it makes perfect sense architecture-wise). It looks like there's no code in xrdp for handling this at all and xrdp only ever checks that the response code is 0 (if I am parsing its logic correctly) and otherwise calls failure. So I think another case needs to be added here to capture request forwarded?

[matt335672]

I think you're right, from the debugging I've done.

I also need a timer event so can pick up if the resize doesn't happen. I'm adding that in at the moment, but it's going to be a bit slow for the rest of the week.

[matt335672]

I've added support for the forwarded resize request to VNC. This seems to work pretty well, but occasionally I get a wayvnc crash which then prevents further access to the session. Depending on how often this happens I'll need to debug it, or add in restart support for wayvnc.

We're going to need the 'forwarded resize request' support in devel I think as well, regardless of what happens here. I'm going to pull the commits for this recent change into a separate PR with test cases which I can test separately. Then I'll rebase this PR on the new devel. I'll also move my test VM for this branch onto Debian trixie.

[matt335672]

The required VNC functionality is now merged into devel, and I've rebased this PR.

[lcniel]

I've added support for the forwarded resize request to VNC. This seems to work pretty well, but occasionally I get a wayvnc crash which then prevents further access to the session. Depending on how often this happens I'll need to debug it, or add in restart support for wayvnc.

We're going to need the 'forwarded resize request' support in devel I think as well, regardless of what happens here. I'm going to pull the commits for this recent change into a separate PR with test cases which I can test separately. Then I'll rebase this PR on the new devel. I'll also move my test VM for this branch onto Debian trixie.

Hi, I tried it out, it works as advertised! Nice work!

I can confirm though that wayvnc will segfault from time to time. I seem to be able to consistently trigger it by resizing the window below c. 860x720 (extrapolating from my monitor resolution). However, if I directly connect at this resolution, it will connect without crashing. In general this is not consistent with how my client (Remmina) normally behaves when connecting to xrdp.

[matt335672]

I think the next step will be for me to move to Trixie, build wayvnc from source and figure out the crash. Until that's fixed we can't use VNC as a production level interface to Wayland.

[lcniel]

I think the next step will be for me to move to Trixie, build wayvnc from source and figure out the crash. Until that's fixed we can't use VNC as a production level interface to Wayland.

I had a bit of a look but I can't seem to get trace-level debug for wayvnc via the sesman.ini settings. I did get this, by resizing back and forth:

From journald:

Aug 25 14:06:37 wayrocky10 xrdp[104579]: [DEBUG] VNC_RESIZE: Sending SetDesktopSize 804x679
Aug 25 14:06:37 wayrocky10 xrdp[104579]: [DEBUG] VNC_RESIZE: VNC matched ExtendedDesktopSize rectangle x=1, y=4 geom=804x679
Aug 25 14:06:37 wayrocky10 xrdp[104579]: [DEBUG] VNC_RESIZE: VNC server resize forwarded
Aug 25 14:06:37 wayrocky10 xrdp[104579]: [INFO ] Layout from ForwardedLayout (geom=804x679 #screens=1) : 0:(804x679+0+0)
Aug 25 14:06:37 wayrocky10 xrdp[104579]: [DEBUG] VNC_RESIZE: VNC matched ExtendedDesktopSize rectangle x=0, y=0 geom=806x1401
Aug 25 14:06:37 wayrocky10 xrdp[104579]: [DEBUG] Skipping RFB_ENC_RAW encoding
Aug 25 14:06:37 wayrocky10 xrdp[104579]: [DEBUG] VNC_RESIZE: Ignored ExtendedDesktopSize 806x1401 x=0 y=0
Aug 25 14:06:37 wayrocky10 kernel: wayvnc[104675]: segfault at 7f19b1d92000 ip 00007f19cc24ebd9 sp 00007f19b2590f78 error 4 in libc.so.6[165bd9,7f19cc111000+150000] like>

From wayvnc debug (snipped)

DEBUG: ../src/main.c: 820: Client resolution changed: 808x679, capturing output HEADLESS-1 which is headless: yes
DEBUG: ../src/server.c: 1673: Client requested resize to 808x679, result: 4
DEBUG: ../src/server.c: 1631: Sending extended desktop resize rect: 808x679
DEBUG: ../src/server.c: 1631: Sending extended desktop resize rect: 676x1183
...
DEBUG: ../src/main.c: 820: Client resolution changed: 814x1401, capturing output HEADLESS-1 which is headless: yes
DEBUG: ../src/server.c: 1673: Client requested resize to 814x1401, result: 4
DEBUG: ../src/server.c: 1631: Sending extended desktop resize rect: 814x1401
DEBUG: ../src/server.c: 1631: Sending extended desktop resize rect: 694x480

I also get some tearing from time to time. It looks to me like screen width is off by some (variable?) amount at some points. I think there might be both that and a race condition going on? I can't easily match up logs since the wayvnc ones don't have timestamps.

EDIT: Another try.

xrdp:

DEBUG] VNC_RESIZE: Sending SetDesktopSize 804x1401
DEBUG] VNC_RESIZE: VNC matched ExtendedDesktopSize rectangle x=1, y=4 geom=804x1401
DEBUG] VNC_RESIZE: VNC server resize forwarded
INFO ] Layout from ForwardedLayout (geom=804x1401 #screens=1) : 0:(804x1401+0+0)
DEBUG] VNC_RESIZE: VNC matched ExtendedDesktopSize rectangle x=0, y=0 geom=1672x1401
DEBUG] Skipping RFB_ENC_RAW encoding
DEBUG] VNC_RESIZE: Ignored ExtendedDesktopSize 1672x1401 x=0 y=0
DEBUG] VNC_RESIZE: VNC matched ExtendedDesktopSize rectangle x=0, y=0 geom=804x1401
DEBUG] Skipping RFB_ENC_RAW encoding
DEBUG] VNC_RESIZE: VNC server forwarded resize complete
INFO ] server_monitor_resize_done: Advancing server monitor resized.
DEBUG] VNC_RESIZE: Read ExtendedDesktopSize 804x1401 x=0 y=0
... (other stuff)
DEBUG] VNC_RESIZE: Sending SetDesktopSize 804x679
DEBUG] Skipping RFB_ENC_RAW encoding
DEBUG] Skipping RFB_ENC_RAW encoding
DEBUG] VNC_RESIZE: VNC matched ExtendedDesktopSize rectangle x=1, y=4 geom=804x679
DEBUG] VNC_RESIZE: VNC server resize forwarded
INFO ] Layout from ForwardedLayout (geom=804x679 #screens=1) : 0:(804x679+0+0)
DEBUG] VNC_RESIZE: VNC matched ExtendedDesktopSize rectangle x=0, y=0 geom=804x1401
DEBUG] Skipping RFB_ENC_RAW encoding
DEBUG] VNC_RESIZE: Ignored ExtendedDesktopSize 804x1401 x=0 y=0
ERROR] VNC server closed connection (segfault)

wayvnc:

DEBUG: ../src/server.c: 1673: Client requested resize to 804x1401, result: 4
DEBUG: ../src/server.c: 1631: Sending extended desktop resize rect: 804x1401
DEBUG: ../src/server.c: 1631: Sending extended desktop resize rect: 1672x1401
DEBUG: ../src/main.c: 1075: Output dimensions changed. Restarting frame capturer...
DEBUG: ../src/buffer.c: 606: Reconfiguring buffer pool
DEBUG: ../src/server.c: 1631: Sending extended desktop resize rect: 804x1401
DEBUG: ../src/server.c: 1631: Sending extended desktop resize rect: 804x1401
DEBUG: ../src/server.c: 1631: Sending extended desktop resize rect: 804x1401
(this is where the xrdp output break is)
DEBUG: ../src/main.c: 820: Client resolution changed: 804x679, capturing output HEADLESS-1 which is headless: yes
DEBUG: ../src/server.c: 1673: Client requested resize to 804x679, result: 4
DEBUG: ../src/server.c: 1631: Sending extended desktop resize rect: 804x679
DEBUG: ../src/server.c: 1631: Sending extended desktop resize rect: 804x1401
DEBUG: ../src/main.c: 1075: Output dimensions changed. Restarting frame capturer...
DEBUG: ../src/buffer.c: 606: Reconfiguring buffer pool
(segfault here)

Sizes seem to match here and I didn't see the same kind of tearing. But I'm not sure wayvnc's messages should look like that? Should it be sending multiple extended desktop sizes that are then ignored?

My vacation starts next week so depending on how things are I might also invest some time into this. At least I can test stuff on RHEL-likes. I don't know if it would be feasible to run valgrind or something to try and catch the segafault. I'm going to guess there's an incorrectly sized buffer or similar at play here.

[matt335672]

I'll try with the latest wayvnc on Trixie when I get it set up. Running with valgrind and/or gdb should be straightforward.

[matt335672]

Force-push to incorporate #3596 and #3597

I've put this onto Trixie now. Running with the stock wayvnc was identical to Xubuntu 25-04 in terms of results.

I've now rebuilt wayvnc from source:-

$ /usr/bin/wayvnc --version
wayvnc: 0.9.1
neatvnc: 0.9.1
aml: 0.3.0

$ ~/wayvnc/build/wayvnc --version
wayvnc: v0.10-dev-f4d0518 (master)
neatvnc: v0.10-dev-4962e0a (master)
aml: v1.0.0-0-g685035c (master)

Using the devel version seems a lot more solid and I've not crashed it yet with resizing. I'll try to work out if a recent change to one of the three products above has fixed the resize issue.

[matt335672]

I've tried rolling back to :

$ ~/wayvnc/build/wayvnc --version
wayvnc: v0.9.1-e4ec935 (HEAD)
neatvnc: v0.9.2-4c37ae9 (HEAD)
aml: v0.3.0-0-gb83f357 (HEAD)

The change is a few fixes to the neatvnc library used by wayvnc

https://github.com/any1/neatvnc/releases/tag/v0.9.2

That seems pretty solid too.

I've rebased on wayvnc 0.9.1 / neatvnc 0.9.5, and I'll continue testing with that for now.

[lcniel]

I've tried rolling back to :

$ ~/wayvnc/build/wayvnc --version
wayvnc: v0.9.1-e4ec935 (HEAD)
neatvnc: v0.9.2-4c37ae9 (HEAD)
aml: v0.3.0-0-gb83f357 (HEAD)

The change is a few fixes to the neatvnc library used by wayvnc

https://github.com/any1/neatvnc/releases/tag/v0.9.2

That seems pretty solid too.

I've rebased on wayvnc 0.9.1 / neatvnc 0.9.5, and I'll continue testing with that for now.

I tested with wayvnc 0.9.1, neatvnc 0.9.5, and a built-from-source aml 0.3.0 (all very simple on Rocky 10, it's enough to just download source RPM:s and replace the version numbers before rpmbuild -ba package.spec) As you say it seems a lot more stable, I can't provoke a crash at all. I feel like there's a slightly higher, barely perceptibly so, latency but this could be my imagination, or maybe even just higher load on my wireguard or something. I think my librdp client flatpak updated yesterday as well so that could also be it. Anyway, at worst that's the price of stability and the performance is certainly more than good enough.

Very promising, all of this!