Test proposed shared buffer PR in JavaFX with VLCJ

[hohonuuli]

JavaFX has a new pull request to support native rendering by supporting WritableImages backed by NIO bytebuffers. This would allow JavaFX to avoid copying data from native buffers to a WritableImage, greatly reducing CPU load. The details of the PR are at https://mail.openjdk.java.net/pipermail/openjfx-dev/2019-June/023347.html.

I thought VLCJ would be provide a good test for this PR.

[caprica]

This is interesting, thanks for the tip!

[caprica]

I had a bit of time to make this work:

https://github.com/caprica/vlcj-javafx/tree/pixelbuffer-test

Feedback welcome, especially if anyone runs performance tests or if a JavaFX expert has any more tips for a more optimal solution.

[hohonuuli]

Awesome! I'm not a JavaFX expert. Still, I hope to carve out a moment tonight to take your test for spin. Thanks for putting that together.

[hohonuuli]

I went to run the demo and I'm getting this compile error in JavaFXDirectRenderingTest:

Error:(224, 86) java: cannot find symbol
  symbol:   method getNativeBuffers()
  location: variable videoSurface of type uk.co.caprica.vlcj.javafx.test.JavaFXDirectRenderingTest.JavaFxVideoSurface

The offending line is:

pixelBuffer = new PixelBuffer(bufferWidth, bufferHeight, videoSurface.getNativeBuffers()[0], pixelFormat);

There's no method getNativeBuffers() ... any tips?

[hohonuuli]

OK making progress, I went to modify the vlcj source to add a method CallbackVideoSurface.getNativeBuffers(). But when I run mvn install I get an error:

Caused by: java.lang.UnsatisfiedLinkError: Unable to load library 'vlc':
dlopen(libvlc.dylib, 9): image not found
dlopen(libvlc.dylib, 9): image not found
Native library (darwin/libvlc.dylib) not found in resource path

I have VLC installed. What's the incantation to get libvlc on the lib path for vlcj's maven build?

[hohonuuli]

Nevermind ... I didn't pay attention that libvlc was only used for tests. I was able to compile with:

mvn install -Dmaven.javadoc.skip=true -Dmaven.test.skip=true

[hohonuuli]

I ran a quick test with the following:

tl;dr: Best rending I've seen from vlcj on a Mac. 👍

I have some 4k ProRes files I'd like to run tests with, but I'm off on vacation for a week. I'll run those tests when I get back.

Video Source

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'V4066_20170822T135738Z_h264.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    creation_time   : 2017-08-22T13:57:38.000000Z
    encoder         : Lavf56.22.100
  Duration: 00:15:00.42, start: 0.000000, bitrate: 26185 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 26175 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
    Metadata:
      creation_time   : 2017-08-22T13:57:38.000000Z
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 3 kb/s (default)
    Metadata:
      creation_time   : 2017-08-22T13:57:38.000000Z
      handler_name    : SoundHandler

JDK

openjdk version "12.0.1" 2019-04-16
OpenJDK Runtime Environment AdoptOpenJDK (build 12.0.1+12)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 12.0.1+12, mixed mode, sharing)

OS

Test profiling

[hohonuuli]

The only change I a had to make to vlcj was:

--- a/src/main/java/uk/co/caprica/vlcj/player/embedded/videosurface/CallbackVideoSurface.java
+++ b/src/main/java/uk/co/caprica/vlcj/player/embedded/videosurface/CallbackVideoSurface.java
@@ -79,26 +85,33 @@ public class CallbackVideoSurface extends VideoSurface {
         libvlc_video_set_callbacks(mediaPlayer.mediaPlayerInstance(), lock, unlock, display, null);
     }

+    public ByteBuffer[] getNativeBuffers() {
+        return nativeBuffers.buffers();
+    }

[hohonuuli]

For grins, I ran a test with the new Shenandoah GC ... note the memory usage!

[caprica]

I'll make a corresponding branch in vlcj some time later today to include that change I forgot, thanks.

And thanks for the feedback.

[caprica]

You need the 4.2.0-SNAPSHOT that (at time of writing) is the current vlcj master.

Use:

mvn -Dmaven.test.skip=true clean install

You must use that snapshot build rather than a release build (as per the comments above).

[caprica]

So that profile chart posted by @hohonuuli shows 9Gb heap usage which is insane, but interestingly the CPU load seems to about 8% on average which is somewhat remarkable.