Skip to content

Conversation

maurerdietmar
Copy link

Add support for the Qemu audio extension.

changes in v2:

  • ability to re-sync by resampling audio
  • move audio code to separate file

pbo-linaro pushed a commit to pbo-linaro/qemu-ci that referenced this pull request Apr 8, 2025
https://lore.kernel.org/qemu-devel/[email protected]

---

From: Dietmar Maurer <[email protected]>
To: [email protected],
	[email protected]
Cc: Dietmar Maurer <[email protected]>
Subject: [PATCH 0/3] Add VNC Open H.264 Encoding
Date: Tue,  8 Apr 2025 09:15:29 +0200
Message-Id: <[email protected]>
X-Mailer: git-send-email 2.39.5
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Host-Lookup-Failed: Reverse DNS lookup failed for 94.136.29.99 (failed)
Received-SPF: none client-ip=94.136.29.99;
 [email protected]; helo=zilli.proxmox.com
X-Spam_score_int: -10
X-Spam_score: -1.1
X-Spam_bar: -
X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, NO_DNS_FOR_FROM=0.001,
 RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001,
 RDNS_NONE=0.793, SPF_HELO_NONE=0.001,
 SPF_NONE=0.001 autolearn=no autolearn_force=no
X-Spam_action: no action
X-BeenThere: [email protected]
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:[email protected]?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:[email protected]>
List-Help: <mailto:[email protected]?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:[email protected]?subject=subscribe>
Errors-To: [email protected]
Sender: [email protected]

As defined by:

https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#open-h-264-encoding

The noVNC HTML application recently added support for this encoding. There is
also an open pull request to add audio support to noVNC:

novnc/noVNC#1952

With that in place, the web based VNC console is good enough to display
a VM showing a video with reasonable bandwidth.

Possible improvements:

- Dynamic switching to/from H264 mode at high change rates
- Support for hardware encoders

We may also extend the RFB Audio protocol with "opus" encoding, because uncompressed
audio need too much bandwidth.

Dietmar Maurer (3):
  new configure option to enable gstreamer
  add vnc h264 encoder
  vnc: h264: send additional frames after the display is clean

 meson.build                   |  10 ++
 meson_options.txt             |   2 +
 scripts/meson-buildoptions.sh |   5 +-
 ui/meson.build                |   1 +
 ui/vnc-enc-h264.c             | 269 ++++++++++++++++++++++++++++++++++
 ui/vnc-jobs.c                 |  49 +++++--
 ui/vnc.c                      |  46 +++++-
 ui/vnc.h                      |  24 +++
 8 files changed, 389 insertions(+), 17 deletions(-)
 create mode 100644 ui/vnc-enc-h264.c

--
2.39.5

Signed-off-by: GitHub Actions Bot <[email protected]>
pbo-linaro pushed a commit to pbo-linaro/qemu-ci that referenced this pull request Apr 10, 2025
https://lore.kernel.org/qemu-devel/[email protected]

---

From: Dietmar Maurer <[email protected]>
To: [email protected],
	[email protected]
Cc: Dietmar Maurer <[email protected]>
Subject: [PATCH v2 0/6] Add VNC Open H.264 Encoding
Date: Thu, 10 Apr 2025 13:22:32 +0200
Message-Id: <[email protected]>
X-Mailer: git-send-email 2.39.5
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Host-Lookup-Failed: Reverse DNS lookup failed for 94.136.29.99 (failed)
Received-SPF: none client-ip=94.136.29.99;
 [email protected]; helo=zilli.proxmox.com
X-Spam_score_int: -10
X-Spam_score: -1.1
X-Spam_bar: -
X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, NO_DNS_FOR_FROM=0.001,
 RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001,
 RDNS_NONE=0.793, SPF_HELO_NONE=0.001,
 SPF_NONE=0.001 autolearn=no autolearn_force=no
X-Spam_action: no action
X-BeenThere: [email protected]
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:[email protected]?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:[email protected]>
List-Help: <mailto:[email protected]?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:[email protected]?subject=subscribe>
Errors-To: [email protected]
Sender: [email protected]

As defined by:

https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#open-h-264-encoding

The noVNC HTML application recently added support for this encoding. There is
also an open pull request to add audio support to noVNC:

novnc/noVNC#1952

With that in place, the web based VNC console is good enough to display
a VM showing a video with reasonable bandwidth.

Possible improvements:

- Dynamic switching to/from H264 mode at high change rates
- Support for hardware encoders

We may also extend the RFB Audio protocol with "opus" encoding, because uncompressed
audio need too much bandwidth.

Changes in v2:

- cleanup: h264: remove wrong libavcodec_ prefix from function names
- search for available h264 encoder, and only enable h264 if a
  encoder is available
- new vnc option to configure h264 at server side

Dietmar Maurer (6):
  new configure option to enable gstreamer
  add vnc h264 encoder
  vnc: h264: send additional frames after the display is clean
  h264: remove wrong libavcodec_ prefix from function names
  h264: search for available h264 encoder
  h264: new vnc option to configure h264 at server side

 meson.build                   |  10 +
 meson_options.txt             |   2 +
 scripts/meson-buildoptions.sh |   5 +-
 ui/meson.build                |   1 +
 ui/vnc-enc-h264.c             | 335 ++++++++++++++++++++++++++++++++++
 ui/vnc-jobs.c                 |  49 +++--
 ui/vnc.c                      |  62 ++++++-
 ui/vnc.h                      |  29 +++
 8 files changed, 476 insertions(+), 17 deletions(-)
 create mode 100644 ui/vnc-enc-h264.c

--
2.39.5

Signed-off-by: GitHub Actions Bot <[email protected]>
Copy link
Contributor

@CendioZeijlon CendioZeijlon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall a nice implementation!

I found that I could make the audio stutter, e.g. when running a video in full screen. But I'm not sure if this is due to the CPU being overloaded or not, the Image would also stutter when this happened.

I would also like to see unittests for your code.

pbo-linaro pushed a commit to pbo-linaro/qemu-ci that referenced this pull request Apr 18, 2025
https://lore.kernel.org/qemu-devel/[email protected]

---

From: Dietmar Maurer <[email protected]>
To: [email protected],
	[email protected]
Cc: Dietmar Maurer <[email protected]>
Subject: [PATCH v3 0/9] Add VNC Open H.264 Encoding
Date: Fri, 18 Apr 2025 13:29:44 +0200
Message-Id: <[email protected]>
X-Mailer: git-send-email 2.39.5
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Host-Lookup-Failed: Reverse DNS lookup failed for 94.136.29.99 (failed)
Received-SPF: none client-ip=94.136.29.99;
 [email protected]; helo=zilli.proxmox.com
X-Spam_score_int: -10
X-Spam_score: -1.1
X-Spam_bar: -
X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, NO_DNS_FOR_FROM=0.001,
 RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001,
 RDNS_NONE=0.793, SPF_HELO_NONE=0.001,
 SPF_NONE=0.001 autolearn=no autolearn_force=no
X-Spam_action: no action
X-BeenThere: [email protected]
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:[email protected]?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:[email protected]>
List-Help: <mailto:[email protected]?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:[email protected]?subject=subscribe>
Errors-To: [email protected]
Sender: [email protected]

As defined by:

https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#open-h-264-encoding

The noVNC HTML application recently added support for this encoding. There is
also an open pull request to add audio support to noVNC:

novnc/noVNC#1952

With that in place, the web based VNC console is good enough to display
a VM showing a video with reasonable bandwidth.

Possible improvements:

- Dynamic switching to/from H264 mode at high change rates
- do not compute all rects in vnc_update_client to reduce CPU load

We may also extend the RFB Audio protocol with "opus" encoding, because uncompressed
audio need too much bandwidth.

Changes in v3:

- add license header
- sqash patch to remove libavcodec prefix
- use gst_clear_object and goto error
- use single g_object_set
- g_autoptr/g_new0
- document vnc_h264_send_framebuffer_update returnm value
- avoid mixed declarations
- use loop to retrieve samples
- initialize gst during argument processing
- add hardware encoders

Changes in v2:

- cleanup: h264: remove wrong libavcodec_ prefix from function names
- search for available h264 encoder, and only enable h264 if a
  encoder is available
- new vnc option to configure h264 at server side

Dietmar Maurer (9):
  new configure option to enable gstreamer
  add vnc h264 encoder
  vnc: h264: send additional frames after the display is clean
  h264: search for available h264 encoder
  h264: new vnc option to configure h264 at server side
  h264: add hardware encoders
  h264: do not reduce vnc update speed while we have an active h264
    stream
  vnc: initialize gst during argument processing
  h264: register shutdown notifiers, stop pipeline in
    destroy_encoder_context

 meson.build                   |  10 +
 meson_options.txt             |   2 +
 scripts/meson-buildoptions.sh |   5 +-
 system/vl.c                   |   8 +
 ui/meson.build                |   1 +
 ui/vnc-enc-h264.c             | 401 ++++++++++++++++++++++++++++++++++
 ui/vnc-jobs.c                 |  49 +++--
 ui/vnc.c                      |  63 +++++-
 ui/vnc.h                      |  30 +++
 9 files changed, 551 insertions(+), 18 deletions(-)
 create mode 100644 ui/vnc-enc-h264.c

--
2.39.5

Signed-off-by: GitHub Actions Bot <[email protected]>
@maurerdietmar
Copy link
Author

I am not really sure if we should rename class Audio to QemuAudio, because we can easily add support for other formats in the future. If you really want to rename the class, should we also rename the file to "qemu-audio.js"?

@berrange berrange mentioned this pull request Apr 24, 2025
pbo-linaro pushed a commit to pbo-linaro/qemu-ci that referenced this pull request Apr 28, 2025
https://lore.kernel.org/qemu-devel/[email protected]

---

From: Dietmar Maurer <[email protected]>
To: [email protected],
	[email protected]
Cc: Dietmar Maurer <[email protected]>
Subject: [PATCH v4 0/8] Add VNC Open H.264 Encoding
Date: Mon, 28 Apr 2025 10:03:28 +0200
Message-Id: <[email protected]>
X-Mailer: git-send-email 2.39.5
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Host-Lookup-Failed: Reverse DNS lookup failed for 94.136.29.99 (failed)
Received-SPF: none client-ip=94.136.29.99;
 [email protected]; helo=zilli.proxmox.com
X-Spam_score_int: -10
X-Spam_score: -1.1
X-Spam_bar: -
X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, NO_DNS_FOR_FROM=0.001,
 RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001,
 RDNS_NONE=0.793, SPF_HELO_NONE=0.001,
 SPF_NONE=0.001 autolearn=no autolearn_force=no
X-Spam_action: no action
X-BeenThere: [email protected]
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:[email protected]?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:[email protected]>
List-Help: <mailto:[email protected]?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:[email protected]?subject=subscribe>
Errors-To: [email protected]
Sender: [email protected]

As defined by:

https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#open-h-264-encoding

The noVNC HTML application recently added support for this encoding. There is
also an open pull request to add audio support to noVNC:

novnc/noVNC#1952

With that in place, the web based VNC console is good enough to display
a VM showing a video with reasonable bandwidth.

Possible improvements:

- Decide whether we should allow --gst-* arguments on the command line
- Dynamic switching to/from H264 mode at high change rates

We may also extend the RFB Audio protocol with "opus" encoding, because uncompressed
audio need too much bandwidth.

Changes in v4:

- style fixes and cleanup (as suggested by Marc and Daniel)
- use SPDX-License-Identifier for new file
- protect everything with #ifdef CONFIG_GSTREAMER
- simplify vnc_refresh as suggested by Daniel (avoid populating
  the rectangle array, do not use vs after it might be freed)
- make "h264" a boolean option
- add new "h264-encoders" options (colon separated)
- add a cleanup path for vnc instead of using shutdown notifiers.

Changes in v3:

- add license header
- sqash patch to remove libavcodec prefix
- use gst_clear_object and goto error
- use single g_object_set
- g_autoptr/g_new0
- document vnc_h264_send_framebuffer_update returnm value
- avoid mixed declarations
- use loop to retrieve samples
- initialize gst during argument processing
- add hardware encoders

Changes in v2:

- cleanup: h264: remove wrong libavcodec_ prefix from function names
- search for available h264 encoder, and only enable h264 if a
  encoder is available
- new vnc option to configure h264 at server side

Dietmar Maurer (8):
  new configure option to enable gstreamer
  vnc: initialize gst during argument processing
  add vnc h264 encoder
  vnc: h264: send additional frames after the display is clean
  h264: search for available h264 encoder
  h264: new vnc options to configure h264 at server side
  h264: add hardware encoders
  h264: stop gstreamer pipeline before destroying, cleanup on exit

 include/ui/console.h          |   1 +
 meson.build                   |  10 +
 meson_options.txt             |   2 +
 scripts/meson-buildoptions.sh |   3 +
 system/runstate.c             |   2 +
 system/vl.c                   |   8 +
 ui/meson.build                |   1 +
 ui/vnc-enc-h264.c             | 358 ++++++++++++++++++++++++++++++++++
 ui/vnc-jobs.c                 |  53 +++--
 ui/vnc.c                      |  80 +++++++-
 ui/vnc.h                      |  29 +++
 11 files changed, 529 insertions(+), 18 deletions(-)
 create mode 100644 ui/vnc-enc-h264.c

--
2.39.5

Signed-off-by: GitHub Actions Bot <[email protected]>
@CendioZeijlon
Copy link
Contributor

Please resolve the comments you think are fixed, and edit your commits to what you would want the end result to look like and do a push with --force.

pbo-linaro pushed a commit to pbo-linaro/qemu-ci that referenced this pull request Apr 30, 2025
https://lore.kernel.org/qemu-devel/[email protected]

---

From: Dietmar Maurer <[email protected]>
To: [email protected],
	[email protected]
Cc: Dietmar Maurer <[email protected]>
Subject: [PATCH v5 0/7] Add VNC Open H.264 Encoding
Date: Wed, 30 Apr 2025 09:25:17 +0200
Message-Id: <[email protected]>
X-Mailer: git-send-email 2.39.5
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Host-Lookup-Failed: Reverse DNS lookup failed for 94.136.29.99 (failed)
Received-SPF: none client-ip=94.136.29.99;
 [email protected]; helo=zilli.proxmox.com
X-Spam_score_int: -10
X-Spam_score: -1.1
X-Spam_bar: -
X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, NO_DNS_FOR_FROM=0.001,
 RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001,
 RDNS_NONE=0.793, SPF_HELO_NONE=0.001,
 SPF_NONE=0.001 autolearn=no autolearn_force=no
X-Spam_action: no action
X-BeenThere: [email protected]
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
 <mailto:[email protected]?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:[email protected]>
List-Help: <mailto:[email protected]?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
 <mailto:[email protected]?subject=subscribe>
Errors-To: [email protected]
Sender: [email protected]

https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#open-h-264-encoding

The noVNC HTML application recently added support for this encoding. There is
also an open pull request to add audio support to noVNC:

novnc/noVNC#1952

With that in place, the web based VNC console is good enough to display
a VM showing a video with reasonable bandwidth.

Possible improvements:

- Dynamic switching to/from H264 mode at high change rates

We may also extend the RFB Audio protocol with "opus" encoding, because uncompressed
audio need too much bandwidth.

Changes in V5:

- Do not allow --gst-* arguments on the command line (initialize gst
  in vnc_init_func)
- fix broken build by moving cleanup code to vl.c

Changes in v4:

- style fixes and cleanup (as suggested by Marc and Daniel)
- use SPDX-License-Identifier for new file
- protect everything with #ifdef CONFIG_GSTREAMER
- simplify vnc_refresh as suggested by Daniel (avoid populating
  the rectangle array, do not use vs after it might be freed)
- make "h264" a boolean option
- add new "h264-encoders" options (colon separated)
- add a cleanup path for vnc instead of using shutdown notifiers.

Changes in v3:

- add license header
- sqash patch to remove libavcodec prefix
- use gst_clear_object and goto error
- use single g_object_set
- g_autoptr/g_new0
- document vnc_h264_send_framebuffer_update returnm value
- avoid mixed declarations
- use loop to retrieve samples
- initialize gst during argument processing
- add hardware encoders

Changes in v2:

- cleanup: h264: remove wrong libavcodec_ prefix from function names
- search for available h264 encoder, and only enable h264 if a
  encoder is available
- new vnc option to configure h264 at server side

Dietmar Maurer (7):
  new configure option to enable gstreamer
  add vnc h264 encoder
  vnc: h264: send additional frames after the display is clean
  h264: search for available h264 encoder
  h264: new vnc options to configure h264 at server side
  h264: add hardware encoders
  h264: stop gstreamer pipeline before destroying, cleanup on exit

 include/system/system.h       |   1 +
 include/ui/console.h          |   1 +
 meson.build                   |  10 +
 meson_options.txt             |   2 +
 scripts/meson-buildoptions.sh |   3 +
 system/runstate.c             |   2 +
 system/vl.c                   |   7 +
 ui/meson.build                |   1 +
 ui/vnc-enc-h264.c             | 357 ++++++++++++++++++++++++++++++++++
 ui/vnc-jobs.c                 |  53 +++--
 ui/vnc.c                      |  84 +++++++-
 ui/vnc.h                      |  29 +++
 12 files changed, 532 insertions(+), 18 deletions(-)
 create mode 100644 ui/vnc-enc-h264.c

--
2.39.5

Signed-off-by: GitHub Actions Bot <[email protected]>
@maurerdietmar maurerdietmar force-pushed the qemu-audio-support-v2 branch from d274eaa to 3721da9 Compare May 7, 2025 09:22
- use better normalization: (value - 32768) / 32768.0;
- use DataView to access little-endian 16bit raw audio data

Signed-off-by: Dietmar Maurer <[email protected]>
@liyimeng
Copy link

Great!

@CendioZeijlon
Copy link
Contributor

Apologies for the lack of activity in this PR on my part, I'll try to pick up the slack :)

I left this in a review comment, maybe it was missed:

I've given this a bit more thought, and I'll have to go back to what I said here. We don't want to add new things to the RFB class/API if we don't have to.

Ideally, we would like to ask the system/browser running the client for it's preferred audio settings by some means. But the settings you had there before are good enough as a starting point.

Let's also assume that users want audio enabled, such that we don't have to add settings for toggling audio on and off etc. This should mean that you don't have to change the RFB-class API by adding e.g. an enable_audio()-function.

@maurerdietmar
Copy link
Author

Let's also assume that users want audio enabled, such that we don't have to add settings for toggling audio on and off etc. This should mean that you don't have to change the RFB-class API by adding e.g. an enable_audio()-function.

Considering audio is not compressed (currently), this can lead to high bandwidth usage. So IMHO it a good choice to make this configurable...

@CendioZeijlon
Copy link
Contributor

Considering audio is not compressed (currently), this can lead to high bandwidth usage. So IMHO it a good choice to make this configurable...

@CendioOssman, do you have any input here? I know we prefer to avoid visible changes to the RFB object, but @maurerdietmar raises a good point.

@CendioOssman
Copy link
Member

VNC by its very nature already assumes a decent availability of bandwidth, but I suppose in some cases audio could be the tipping point.

Make sure it is a property, though, like the other settings.

@liyimeng
Copy link

Let's also assume that users want audio enabled, such that we don't have to add settings for toggling audio on and off etc. This should mean that you don't have to change the RFB-class API by adding e.g. an enable_audio()-function.

Considering audio is not compressed (currently), this can lead to high bandwidth usage. So IMHO it a good choice to make this configurable...

Is h.264 handle audio compression together with video? when not take care of audio and video via h.264?

@kroese
Copy link

kroese commented Sep 15, 2025

Can this be merged? Or what is holding it back?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants