noVNC stops working when lossy compression is enabled in qemu

[basilky]

Describe the bug
Qemu has a flag to enable or disable lossy compression. noVNC works without any issue when this flag is set to off(default). But when lossy flag enabled, noVNC screen goes blank/gets stuck as soon as the guest os completes boot and loads desktop. Disconnecting and reconnecting doesn't fix this issue. No relevant logs are printed on the noVNC console.

Initially I thought this is an issue with qemu, but qemu vnc screen is connecting without any issue from tigervnc.

To Reproduce
Steps to reproduce the behavior:

1. Start guest os: I used Linux Mint Debian Edition
   qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -m 2000 -cdrom lmde-5-cinnamon-64bit.iso -vnc :0,lossy=on
2. Start novnc: ./utils/novnc_proxy --vnc localhost:5900
3. Open the URL given by noVNC and wait till the guest os completes boot.
4. noVNC screen gets stuck/blank

Expected behavior
noVNC should show the guest screen

Client

• OS: Qemu Host: Linux Mint 21.1, guest: LMDE
• Browser: both Firefox & Chrome
• Browser version: latest

Server (please complete the following information):

• noVNC version: latest(8decca7)
• VNC server: Qemu 6.2.0
• WebSocket proxy: websockify

[CendioOssman]

It's very intriguing that a reconnect doesn't help. That usually resets any protocol issues.

Do you think you could do a recording of the data stream on a reconnect and attach here? There are flags to websockify to enable that.

[basilky]

@CendioOssman Recorded the stream for 5-6 seconds after reconnect
record.txt

[CendioOssman]

Thank you for the recording. I can see now see the issue here, and I get this error from noVNC in the browser console:

Error decoding rect: Error: Gradient filter not implemented

So we have a clear reason for the failure.

It's a bit unexpected given the setting you've described, though. The gradient filter is lossless, so it should be more used when lossy compression is disabled, not enabled.

The gradient filter is rarely used in current VNC servers, so noVNC never got around to implementing it. QEMU is popular with noVNC, though, so I'm surprised that this hasn't surfaced earlier.

[samhed]

This should be fixed as of #1855, @basilky mind testing?

[basilky]

@samhed I tested with the same setup and the latest noVNC is working. Thanks for the fix @mentha