NodeMaterialObserver: Force refresh for video textures.

[Mugen87]

Related issue: #31330 (comment)

Description

Using video textures in WebGPU seems more complicated than in WebGL since it is not possible to share an external texture. We have to recreate the bind group and the external texture for each render item. For material properties, we can enforce this in NodeMaterialObserver.

However, if video textures are used as spot light maps, there is no obvious way to enforce a refresh. Hence, I have removed the video texture from the webgpu_lights_projector demo. I think we can only properly support them on material level. And even then they are a bit problematic for performance in WebGPU since we can't share bindings in the same way like with normal textures.

I suggest to state in the documentation that video textures are not supported in SpotLight.map.

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	338.27 78.93	338.27 78.93	+0 B +0 B
WebGPU	558.52 154.58	558.88 154.69	+361 B +105 B
WebGPU Nodes	557.45 154.37	557.81 154.47	+361 B +107 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	469.61 113.66	469.61 113.66	+0 B +0 B
WebGPU	634.18 171.62	634.54 171.73	+361 B +113 B
WebGPU Nodes	589.31 160.95	589.67 161.11	+361 B +160 B

[sunag]

I'd like to analyze this in more detail. Currently we can take a uniform to an isolated group using uniform().setGroup() then we could share that group between materials using sharedUniformGroup(). But I didn't do the tests using VideoTexture.

[Mugen87]

I'm not sure this fixes the issue.

What irritates me is that the scope/lifetime of an external texture is so short. I've assumed calling importExternalTexture() and using it with a new bind group is valid until the current requestVideoFrameCallback() ends. I was surprised to see the WebGPU warnings during a single render call.

@greggman Can you enlighten us, please? 😇

In the fiddle, we use the same external texture and bind group for two plane meshes. The browser complains now with the warning: "External texture [ExternalTexture (unlabeled)] used in a submit is not active.".

https://jsfiddle.net/86t42dwk/

[greggman]

Without looking, importExternalTexture is valid until the task it's called in exits, regardless of which task. In other wrods, if you call it in requestAnimationFrame it's valid until your requestAnmimationFrame callback returns. If you call it in requestVideoFrameCallback it's valid until your requestVideoFrameCallback returns. If you call it in setTimeout it's valid until your setTimeout callback returns. If you call it inside mouseevent, it's valid until your mouseevent handler returns, etc...

[greggman]

And to be more clear, you must call importExternalTexture, put it in a bindGroup, use it in some pass, and call submit with the encoder that uses it, all in the same task.

[greggman]

Also, not that this explains anything but it seems every other frame the code calls importExternalTexture twice?

https://jsfiddle.net/1s97uLj2/

[Mugen87]

Okay, things are more clear now.

A new video frame is not available in every frame. When no new video frame is ready, the material observer does not trigger a refresh since it thinks the video texture has not changed. However, that will not trigger the required call for importExternalTexture() and creation of a fresh bind group. The PR is fixing exactly this issue.

There is one thing we can optimize: Use importExternalTexture() just once per frame for each video texture.

[Mugen87]

@sunag External textures are now cached per frame id. I've also made sure the video texture refresh is only forced for the WebGPU backend.

I'm curious to see if you can improve the approach somehow via sharedUniformGroup().

[greggman]

It sounds like you solved the issue?

One more caveat. While the external texture needs to be used in the same task, if you call importExternalTexture you might get back the same texture as before, in which case you don't need to create a new bind group.

In other words, in pseudo code

   let prevTexture;
   let bindGroup;

   rAF/rVFC/setTimeout/mouseEvent/etc...
     const texture = device.importExternalTexture(...);
     if (texture !== prevTexture) {
       prevTexture = texture;
       bindGroup = device.createBindGroup({
         entries: [
           ...
           { ..., resource: texture },
           ...
         ],
       ]);
     }

I don't feel like this really matters but just for completeness it seemed good not to leave it out.

[Mugen87]

It sounds like you solved the issue?

Yes, I think it's just a matter of how we organize our code now^^. The importExternalTexture() counter from the fiddle was very helpful, though 👍 .

[Mugen87]

Closing in favor of #31416.

[ycw]

While the external texture needs to be used in the same task, if you call importExternalTexture you might get back the same texture as before

FWIW, this behavior is implementation-dependent

[greggman]

While the external texture needs to be used in the same task, if you call importExternalTexture you might get back the same texture as before

FWIW, this behavior is implementation-dependent

That's the definition of "might get back the same texture".

[ycw]

I just pointed out how the might came from. That being said, I assumed #31416 had guarded from those possible errors like GPUValidationError caused by that reason.

[greggman]

guarded from those possible errors like GPUValidationError caused by that reason.

What validation errors? I guess assuming that you get back the same texture? Yes, it would be good to guard against that.

[ycw]

Let's break it down, renderer.render() could call _renderScene() twice, where _renderScene honors scene.onBeforeRender and scne.onAfterRender protocols, i.e. videoEl.currentTime could be set in those handlers in user code, this could cause videoEl.readyState to be HTMLMediaElement.HAVE_METADATA | HTMLMediaElement.HAVE_NOTHING during the second _renderScene.

If implementations chose to not reuse the previous imported external texture, it'll go through a validation procedure for checking the usability of the videoEl, if videoEl.readyState is either HTMLMediaElement.HAVE_METADATA | HTMLMediaElement.HAVE_NOTHING, implementations will generate GPUValidationError, and return an invalidated GPUExternalTexture.

I didn't fully review #31416, I just assumed that it guarded this edge case.

[greggman]

If implementations chose to not reuse the previous imported external texture

Just to be clear, you can not use a previously imported external texture (in a different task). you can only reuse a bindGroup that contains a previously imported external texture, if you call importExternalTexture again and it turns the same texture. The act of calling importExternalTexture again, either (a) gets you a new texture or (b) unexpires the already expired existing texture. In either case, you must call importExternalTexture. The only optimization you can make is whether or not to make a new bindGroup.

As for checking for usability of the texture, you might want to take a look at the webgpu-sample video sample. It doesn't check things like videoEl.readyState etc.... All it does is await on video.play().

Effectively

const video = const video = document.createElement('video');

let videoCanBeUsed = false;
async function playVideo(url) {
  videoCanBeUsed = false;
  video.src = url;
  await video.play();
  videoCanBeUsed = true;
}

playVideo('some-url-to-a-video');  // call this anytime you want with a new URL.

function render() {
  if (videoCanBeUsed) {
    ...use the video... (call importExternalTexture or copyExternalImageToTexture or gl.texImage2D)
  }
  requestAnimationFrame(render);
}
requestAnimationFrame(render);

I think, all those old methods of checking events and properties have been deprecated.

[ycw]

Thanks for the informative response, I should clarify that implemenations is referring to user agents, not library, there would be c) an invalidated GPUExternalTexture that library should aware of