-
Notifications
You must be signed in to change notification settings - Fork 2.5k
Qemu audio support v2 #1952
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Qemu audio support v2 #1952
Conversation
https://lore.kernel.org/qemu-devel/[email protected] --- From: Dietmar Maurer <[email protected]> To: [email protected], [email protected] Cc: Dietmar Maurer <[email protected]> Subject: [PATCH 0/3] Add VNC Open H.264 Encoding Date: Tue, 8 Apr 2025 09:15:29 +0200 Message-Id: <[email protected]> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Host-Lookup-Failed: Reverse DNS lookup failed for 94.136.29.99 (failed) Received-SPF: none client-ip=94.136.29.99; [email protected]; helo=zilli.proxmox.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, NO_DNS_FOR_FROM=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: [email protected] X-Mailman-Version: 2.1.29 Precedence: list List-Id: <qemu-devel.nongnu.org> List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>, <mailto:[email protected]?subject=unsubscribe> List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel> List-Post: <mailto:[email protected]> List-Help: <mailto:[email protected]?subject=help> List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>, <mailto:[email protected]?subject=subscribe> Errors-To: [email protected] Sender: [email protected] As defined by: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#open-h-264-encoding The noVNC HTML application recently added support for this encoding. There is also an open pull request to add audio support to noVNC: novnc/noVNC#1952 With that in place, the web based VNC console is good enough to display a VM showing a video with reasonable bandwidth. Possible improvements: - Dynamic switching to/from H264 mode at high change rates - Support for hardware encoders We may also extend the RFB Audio protocol with "opus" encoding, because uncompressed audio need too much bandwidth. Dietmar Maurer (3): new configure option to enable gstreamer add vnc h264 encoder vnc: h264: send additional frames after the display is clean meson.build | 10 ++ meson_options.txt | 2 + scripts/meson-buildoptions.sh | 5 +- ui/meson.build | 1 + ui/vnc-enc-h264.c | 269 ++++++++++++++++++++++++++++++++++ ui/vnc-jobs.c | 49 +++++-- ui/vnc.c | 46 +++++- ui/vnc.h | 24 +++ 8 files changed, 389 insertions(+), 17 deletions(-) create mode 100644 ui/vnc-enc-h264.c -- 2.39.5 Signed-off-by: GitHub Actions Bot <[email protected]>
https://lore.kernel.org/qemu-devel/[email protected] --- From: Dietmar Maurer <[email protected]> To: [email protected], [email protected] Cc: Dietmar Maurer <[email protected]> Subject: [PATCH v2 0/6] Add VNC Open H.264 Encoding Date: Thu, 10 Apr 2025 13:22:32 +0200 Message-Id: <[email protected]> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Host-Lookup-Failed: Reverse DNS lookup failed for 94.136.29.99 (failed) Received-SPF: none client-ip=94.136.29.99; [email protected]; helo=zilli.proxmox.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, NO_DNS_FOR_FROM=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: [email protected] X-Mailman-Version: 2.1.29 Precedence: list List-Id: <qemu-devel.nongnu.org> List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>, <mailto:[email protected]?subject=unsubscribe> List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel> List-Post: <mailto:[email protected]> List-Help: <mailto:[email protected]?subject=help> List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>, <mailto:[email protected]?subject=subscribe> Errors-To: [email protected] Sender: [email protected] As defined by: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#open-h-264-encoding The noVNC HTML application recently added support for this encoding. There is also an open pull request to add audio support to noVNC: novnc/noVNC#1952 With that in place, the web based VNC console is good enough to display a VM showing a video with reasonable bandwidth. Possible improvements: - Dynamic switching to/from H264 mode at high change rates - Support for hardware encoders We may also extend the RFB Audio protocol with "opus" encoding, because uncompressed audio need too much bandwidth. Changes in v2: - cleanup: h264: remove wrong libavcodec_ prefix from function names - search for available h264 encoder, and only enable h264 if a encoder is available - new vnc option to configure h264 at server side Dietmar Maurer (6): new configure option to enable gstreamer add vnc h264 encoder vnc: h264: send additional frames after the display is clean h264: remove wrong libavcodec_ prefix from function names h264: search for available h264 encoder h264: new vnc option to configure h264 at server side meson.build | 10 + meson_options.txt | 2 + scripts/meson-buildoptions.sh | 5 +- ui/meson.build | 1 + ui/vnc-enc-h264.c | 335 ++++++++++++++++++++++++++++++++++ ui/vnc-jobs.c | 49 +++-- ui/vnc.c | 62 ++++++- ui/vnc.h | 29 +++ 8 files changed, 476 insertions(+), 17 deletions(-) create mode 100644 ui/vnc-enc-h264.c -- 2.39.5 Signed-off-by: GitHub Actions Bot <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall a nice implementation!
I found that I could make the audio stutter, e.g. when running a video in full screen. But I'm not sure if this is due to the CPU being overloaded or not, the Image would also stutter when this happened.
I would also like to see unittests for your code.
https://lore.kernel.org/qemu-devel/[email protected] --- From: Dietmar Maurer <[email protected]> To: [email protected], [email protected] Cc: Dietmar Maurer <[email protected]> Subject: [PATCH v3 0/9] Add VNC Open H.264 Encoding Date: Fri, 18 Apr 2025 13:29:44 +0200 Message-Id: <[email protected]> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Host-Lookup-Failed: Reverse DNS lookup failed for 94.136.29.99 (failed) Received-SPF: none client-ip=94.136.29.99; [email protected]; helo=zilli.proxmox.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, NO_DNS_FOR_FROM=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: [email protected] X-Mailman-Version: 2.1.29 Precedence: list List-Id: <qemu-devel.nongnu.org> List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>, <mailto:[email protected]?subject=unsubscribe> List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel> List-Post: <mailto:[email protected]> List-Help: <mailto:[email protected]?subject=help> List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>, <mailto:[email protected]?subject=subscribe> Errors-To: [email protected] Sender: [email protected] As defined by: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#open-h-264-encoding The noVNC HTML application recently added support for this encoding. There is also an open pull request to add audio support to noVNC: novnc/noVNC#1952 With that in place, the web based VNC console is good enough to display a VM showing a video with reasonable bandwidth. Possible improvements: - Dynamic switching to/from H264 mode at high change rates - do not compute all rects in vnc_update_client to reduce CPU load We may also extend the RFB Audio protocol with "opus" encoding, because uncompressed audio need too much bandwidth. Changes in v3: - add license header - sqash patch to remove libavcodec prefix - use gst_clear_object and goto error - use single g_object_set - g_autoptr/g_new0 - document vnc_h264_send_framebuffer_update returnm value - avoid mixed declarations - use loop to retrieve samples - initialize gst during argument processing - add hardware encoders Changes in v2: - cleanup: h264: remove wrong libavcodec_ prefix from function names - search for available h264 encoder, and only enable h264 if a encoder is available - new vnc option to configure h264 at server side Dietmar Maurer (9): new configure option to enable gstreamer add vnc h264 encoder vnc: h264: send additional frames after the display is clean h264: search for available h264 encoder h264: new vnc option to configure h264 at server side h264: add hardware encoders h264: do not reduce vnc update speed while we have an active h264 stream vnc: initialize gst during argument processing h264: register shutdown notifiers, stop pipeline in destroy_encoder_context meson.build | 10 + meson_options.txt | 2 + scripts/meson-buildoptions.sh | 5 +- system/vl.c | 8 + ui/meson.build | 1 + ui/vnc-enc-h264.c | 401 ++++++++++++++++++++++++++++++++++ ui/vnc-jobs.c | 49 +++-- ui/vnc.c | 63 +++++- ui/vnc.h | 30 +++ 9 files changed, 551 insertions(+), 18 deletions(-) create mode 100644 ui/vnc-enc-h264.c -- 2.39.5 Signed-off-by: GitHub Actions Bot <[email protected]>
I am not really sure if we should rename class Audio to QemuAudio, because we can easily add support for other formats in the future. If you really want to rename the class, should we also rename the file to "qemu-audio.js"? |
https://lore.kernel.org/qemu-devel/[email protected] --- From: Dietmar Maurer <[email protected]> To: [email protected], [email protected] Cc: Dietmar Maurer <[email protected]> Subject: [PATCH v4 0/8] Add VNC Open H.264 Encoding Date: Mon, 28 Apr 2025 10:03:28 +0200 Message-Id: <[email protected]> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Host-Lookup-Failed: Reverse DNS lookup failed for 94.136.29.99 (failed) Received-SPF: none client-ip=94.136.29.99; [email protected]; helo=zilli.proxmox.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, NO_DNS_FOR_FROM=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: [email protected] X-Mailman-Version: 2.1.29 Precedence: list List-Id: <qemu-devel.nongnu.org> List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>, <mailto:[email protected]?subject=unsubscribe> List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel> List-Post: <mailto:[email protected]> List-Help: <mailto:[email protected]?subject=help> List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>, <mailto:[email protected]?subject=subscribe> Errors-To: [email protected] Sender: [email protected] As defined by: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#open-h-264-encoding The noVNC HTML application recently added support for this encoding. There is also an open pull request to add audio support to noVNC: novnc/noVNC#1952 With that in place, the web based VNC console is good enough to display a VM showing a video with reasonable bandwidth. Possible improvements: - Decide whether we should allow --gst-* arguments on the command line - Dynamic switching to/from H264 mode at high change rates We may also extend the RFB Audio protocol with "opus" encoding, because uncompressed audio need too much bandwidth. Changes in v4: - style fixes and cleanup (as suggested by Marc and Daniel) - use SPDX-License-Identifier for new file - protect everything with #ifdef CONFIG_GSTREAMER - simplify vnc_refresh as suggested by Daniel (avoid populating the rectangle array, do not use vs after it might be freed) - make "h264" a boolean option - add new "h264-encoders" options (colon separated) - add a cleanup path for vnc instead of using shutdown notifiers. Changes in v3: - add license header - sqash patch to remove libavcodec prefix - use gst_clear_object and goto error - use single g_object_set - g_autoptr/g_new0 - document vnc_h264_send_framebuffer_update returnm value - avoid mixed declarations - use loop to retrieve samples - initialize gst during argument processing - add hardware encoders Changes in v2: - cleanup: h264: remove wrong libavcodec_ prefix from function names - search for available h264 encoder, and only enable h264 if a encoder is available - new vnc option to configure h264 at server side Dietmar Maurer (8): new configure option to enable gstreamer vnc: initialize gst during argument processing add vnc h264 encoder vnc: h264: send additional frames after the display is clean h264: search for available h264 encoder h264: new vnc options to configure h264 at server side h264: add hardware encoders h264: stop gstreamer pipeline before destroying, cleanup on exit include/ui/console.h | 1 + meson.build | 10 + meson_options.txt | 2 + scripts/meson-buildoptions.sh | 3 + system/runstate.c | 2 + system/vl.c | 8 + ui/meson.build | 1 + ui/vnc-enc-h264.c | 358 ++++++++++++++++++++++++++++++++++ ui/vnc-jobs.c | 53 +++-- ui/vnc.c | 80 +++++++- ui/vnc.h | 29 +++ 11 files changed, 529 insertions(+), 18 deletions(-) create mode 100644 ui/vnc-enc-h264.c -- 2.39.5 Signed-off-by: GitHub Actions Bot <[email protected]>
Please resolve the comments you think are fixed, and edit your commits to what you would want the end result to look like and do a push with |
https://lore.kernel.org/qemu-devel/[email protected] --- From: Dietmar Maurer <[email protected]> To: [email protected], [email protected] Cc: Dietmar Maurer <[email protected]> Subject: [PATCH v5 0/7] Add VNC Open H.264 Encoding Date: Wed, 30 Apr 2025 09:25:17 +0200 Message-Id: <[email protected]> X-Mailer: git-send-email 2.39.5 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Host-Lookup-Failed: Reverse DNS lookup failed for 94.136.29.99 (failed) Received-SPF: none client-ip=94.136.29.99; [email protected]; helo=zilli.proxmox.com X-Spam_score_int: -10 X-Spam_score: -1.1 X-Spam_bar: - X-Spam_report: (-1.1 / 5.0 requ) BAYES_00=-1.9, NO_DNS_FOR_FROM=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_NONE=0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: [email protected] X-Mailman-Version: 2.1.29 Precedence: list List-Id: <qemu-devel.nongnu.org> List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>, <mailto:[email protected]?subject=unsubscribe> List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel> List-Post: <mailto:[email protected]> List-Help: <mailto:[email protected]?subject=help> List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>, <mailto:[email protected]?subject=subscribe> Errors-To: [email protected] Sender: [email protected] https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#open-h-264-encoding The noVNC HTML application recently added support for this encoding. There is also an open pull request to add audio support to noVNC: novnc/noVNC#1952 With that in place, the web based VNC console is good enough to display a VM showing a video with reasonable bandwidth. Possible improvements: - Dynamic switching to/from H264 mode at high change rates We may also extend the RFB Audio protocol with "opus" encoding, because uncompressed audio need too much bandwidth. Changes in V5: - Do not allow --gst-* arguments on the command line (initialize gst in vnc_init_func) - fix broken build by moving cleanup code to vl.c Changes in v4: - style fixes and cleanup (as suggested by Marc and Daniel) - use SPDX-License-Identifier for new file - protect everything with #ifdef CONFIG_GSTREAMER - simplify vnc_refresh as suggested by Daniel (avoid populating the rectangle array, do not use vs after it might be freed) - make "h264" a boolean option - add new "h264-encoders" options (colon separated) - add a cleanup path for vnc instead of using shutdown notifiers. Changes in v3: - add license header - sqash patch to remove libavcodec prefix - use gst_clear_object and goto error - use single g_object_set - g_autoptr/g_new0 - document vnc_h264_send_framebuffer_update returnm value - avoid mixed declarations - use loop to retrieve samples - initialize gst during argument processing - add hardware encoders Changes in v2: - cleanup: h264: remove wrong libavcodec_ prefix from function names - search for available h264 encoder, and only enable h264 if a encoder is available - new vnc option to configure h264 at server side Dietmar Maurer (7): new configure option to enable gstreamer add vnc h264 encoder vnc: h264: send additional frames after the display is clean h264: search for available h264 encoder h264: new vnc options to configure h264 at server side h264: add hardware encoders h264: stop gstreamer pipeline before destroying, cleanup on exit include/system/system.h | 1 + include/ui/console.h | 1 + meson.build | 10 + meson_options.txt | 2 + scripts/meson-buildoptions.sh | 3 + system/runstate.c | 2 + system/vl.c | 7 + ui/meson.build | 1 + ui/vnc-enc-h264.c | 357 ++++++++++++++++++++++++++++++++++ ui/vnc-jobs.c | 53 +++-- ui/vnc.c | 84 +++++++- ui/vnc.h | 29 +++ 12 files changed, 532 insertions(+), 18 deletions(-) create mode 100644 ui/vnc-enc-h264.c -- 2.39.5 Signed-off-by: GitHub Actions Bot <[email protected]>
Signed-off-by: Dietmar Maurer <[email protected]>
Signed-off-by: Dietmar Maurer <[email protected]>
Signed-off-by: Dietmar Maurer <[email protected]>
d274eaa
to
3721da9
Compare
- use better normalization: (value - 32768) / 32768.0; - use DataView to access little-endian 16bit raw audio data Signed-off-by: Dietmar Maurer <[email protected]>
Great! |
Apologies for the lack of activity in this PR on my part, I'll try to pick up the slack :) I left this in a review comment, maybe it was missed:
Let's also assume that users want audio enabled, such that we don't have to add settings for toggling audio on and off etc. This should mean that you don't have to change the RFB-class API by adding e.g. an enable_audio()-function. |
Considering audio is not compressed (currently), this can lead to high bandwidth usage. So IMHO it a good choice to make this configurable... |
@CendioOssman, do you have any input here? I know we prefer to avoid visible changes to the RFB object, but @maurerdietmar raises a good point. |
VNC by its very nature already assumes a decent availability of bandwidth, but I suppose in some cases audio could be the tipping point. Make sure it is a property, though, like the other settings. |
Is h.264 handle audio compression together with video? when not take care of audio and video via h.264? |
Can this be merged? Or what is holding it back? |
Add support for the Qemu audio extension.
changes in v2: