xorgrdp: XF86VidModeSetGamma() -> BadValue

[jknockel]

xrdp version

0.9.17

Detailed xrdp version, build options

xrdp 0.9.17
  A Remote Desktop Protocol Server.
  Copyright (C) 2004-2020 Jay Sorg, Neutrino Labs, and all contributors.
  See https://github.com/neutrinolabs/xrdp for more information.

  Configure options:
      --enable-ipv6
      --enable-jpeg
      --enable-fuse
      --enable-rfxcodec
      --enable-opus
      --enable-painter
      --enable-vsock
      --build=x86_64-linux-gnu
      --prefix=/usr
      --includedir=${prefix}/include
      --mandir=${prefix}/share/man
      --infodir=${prefix}/share/info
      --sysconfdir=/etc
      --localstatedir=/var
      --disable-silent-rules
      --libdir=${prefix}/lib/x86_64-linux-gnu
      --libexecdir=${prefix}/lib/x86_64-linux-gnu
      --disable-maintainer-mode
      --disable-dependency-tracking
      --with-socketdir=/run/xrdp/sockdir
      build_alias=x86_64-linux-gnu
      CFLAGS=-g -O2 -ffile-prefix-map=/build/xrdp-Doma5Z/xrdp-0.9.17=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security 
      LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed
      CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2 -Wno-error=deprecated-declarations
      PKG_CONFIG_PATH=/build/xrdp-Doma5Z/xrdp-0.9.17/pkgconfig

  Compiled with OpenSSL 3.0.2 15 Mar 2022

Operating system & version

Ubuntu 22.04

Installation method

dnf / apt / zypper / pkg / etc

Which backend do you use?

xorgxrdp

What desktop environment do you use?

gnome-flashback

Environment xrdp running on

Physical or VM

What's your client?

No response

Area(s) with issue?

Graphic glitches, Other

Steps to reproduce

Compile and execute the following program in an xorgrdp session:

gamma.c.zip

✔️ Expected Behavior

Either the XF86VidMode extension not to be present or for the call to XF86VidModeSetGamma() to not deliver an asynchronous BadValue error.

❌ Actual Behavior

XF86VidModeSetGamma() mode crashes the program with a BadValue error.

Anything else?

Many X screensaver programs use XF86VidModeSetGamma() calls to implement a fade-to-black effect before starting the screensaver, but these appear to crash in an xorgrdp session with an asynchronous BadValue error. I couldn't find anything in terms of a specification on XF86VidModeSetGamma(), but these programs are written to not guard against an asynchronous error from calling XF86VidModeSetGamma() and, since they use Xlib, crash when they receive it. However, before calling XF86VidModeSetGamma(), they do check for the presence of the XF86VidMode extension and that it is at least version 2.0, but both of these conditions appear true in an xorgrdp session, and so they fatally call XF86VidModeSetGamma() anyways.

[matt335672]

Thanks for this @jknockel

I'm moving this to the xorgxrdp repo where it can be addressed.

[matt335672]

BTW this is a great fault report - we don't get many test programs submitted.

By default the TigerVNC server (i.e. Xvnc) does not provide this extension.

Here's a bit of analysis:-

The current implementation of the server-side of XF86VidModeSetGamma() can be found here:-

https://cgit.freedesktop.org/xorg/xserver/tree/Xext/vidmode.c#n1391

From that we can see that for a virtual X server, the BadValue is going to be pretty inevitable.

The function pointer above is set up here for each screen:-

https://cgit.freedesktop.org/xorg/xserver/tree/hw/xfree86/common/xf86VidMode.c#n396

That's called from here:-

https://cgit.freedesktop.org/xorg/xserver/tree/hw/xfree86/common/xf86VidMode.c#n440

The best thing to do here is probably to disable the extension as it does not seem to server any useful purpose here. It's a fairly old part of Xorg as the name indicates, and it even has its own option DisableVidModeExtension to disable it.

I've raised a PR (above) to do this for review by the team. In the meantime, if you edit /etc/X11/xrdp/xorg.conf on your system and make the change in the PR this should do the trick for you.

Setting the DisableVidModeExtension option results in the internal server function xf86GetVidModeEnabled() returning false. The links above show that prevents the extension being initialised. The option serves no other function.

[jknockel]

Thanks, adding Option "DisableVidModeExtension" "on" to the ServerFlags section prevents the crash on my end as well.