Serializing WebGLRenderTarget/Cube's texture into a data URL

[jsantell]

Currently, serializing a scene with a WebGLRenderTarget/Cube via toJSON() results in a texture object without a base64 URL, since the texture's image property is empty. We should generate a data URL for the target's current buffer for the target's texture.

[jsantell]

Since render targets own a texture, and the textures are used directly in the scene (rather than the render target) and the only reference of texture-to-rendertarget is a WebGLRenderer internal, I think target.texture.target should be introduced, both for developer transparency and for serialization possibilities. This would allow the option to remove the stubbed texture.image object from #16651, or target can be added in addition to the stubbed image object (I don't know all of the details). Another option would be to introduce a new texture type e.g. WebGLRenderTargetTexture that takes a target instead of an image, or Texture can accept a target instead of a texture.

Possibilities:

1. Add a target property on textures owned by render targets.
2. WebGLRenderTargetTexture that takes a target
3. Texture can receive a target instead of image.

[jsantell]

Allowing Texture to take a render target instead of an image is misleading; it would appear that then new Texture(renderTarget) would be valid, but the render target needs to create and manage the texture. Bolting on target to a regular texture would work for serialization, but still that texture cannot stand alone, and isn't obvious looking at the texture that it represents a render target unless you're familiar with how render targets are handled internally.

Looking at a RenderTargetTexture to solve this which seems like the best choice due to the above.

[jsantell]

Have a rough version of this working. Of course this needs a renderer which is a bit different than serializing other scene objects. WebGLRenderTargets are bound to a renderer once rendering once.

Decisions:

• Should this be opt-in? This would add overhead in memory and compute time on serialization, and may not be desired, although serialization isn't something that was ever performant for runtime.
  • If yes, we can have an API on the WebGLRenderTarget's RenderTargetTexture that computes the URL for the render target's current buffer:
    • (1) By returning it from var url = txt.getDataURL(), which would require stitching back manually into a scene export, but clean and isolated
    • (2) By caching it inside the texture for the next serialization, e.g. txt.cacheDataURL() or txt.snapshotTexture() which would populate the next toJSON() call with an appropriate URL. A bit magical, but retains the serialization flow. Memory overhead increase.
  • If no:
    (3) If serializing a DataTexture/RenderTargetTexture, compute it on requested. Most simple.

Note that since we need a renderer, it can be passed into the prep calls in (1) and (2); for (3), it seems difficult to patch in serialization options, but it's possible. I also have a branch that currently implements the renderer dispatching an event ('bind-renderer') when the render target is first set up/rendered on the texture that it listens to, resulting in each RenderTargetTexture having a reference to its corresponding renderer, allowing the serialization in the first place without having to guess which renderer matches the target.

What do you all think? Or is this something that should remain in user land?

[mrdoob]

As discussed offline, I do not think we should serialize WebGLRenderTarget's texture as it's supposed to be dynamic.

However, I do think the devtools could try to render that texture into a canvas so it can be inspected, but that code should be in the devtools instead.