Provide Access to WebXR camera feed when `camera-access` feature is enabled.

[connorgmeehan]

Description

I would like to be able to get a Texture class that references the camera feed from an XR session.

Solution

User API

Proposal would be to be able to request the camera texture from the WebXRManager class, however this glosses over a lot of implementation detail and may not map to the WebXR access patterns very well.

const texture = renderer.xr.getCameraTexture();

---

Proposed Internal changes

I am currently having a stab implementing this, I have little experience with the internals of three.js so everything is open to change.

Would require the creation of a new THREE.js Texture class that receives the underlying WebGLTexture instead of initialising it. Lets call it ExternalTexture. In an ExternalTexture THREE.js is not responsible for creating / deleting the GPU data, only binding it and tracking its state. As this could introduce a lot of footguns it may be best to keep this class for internal use for now.

const glTexture = gl.createTexture();
const texture = new ExternalTexture(glTexture);

Additionally, the texture provided by the WebXR api has some limitations. It is what the WebXR specs call an opaque texture. In summary it is:

• Only a valid texture from within the requestAnimationFrame() callback of the XRSession
• Calling deleteTexture on this texture will result in an INVALID_OPERATION error.
• Must be unbound and detached from all WebGLShader objects at the end of the requestAnimationFrame()
• Changes to dimensions or format is not allowed and any gl call that could change these parameters will fail with an INVALID_OPERATION error.

This will probably require a second class that extends from ExternalTexture, lets call it ExternalOpaqueTexture. To navigate the requirement that the texture is only valid from within an XRSession's requestAnimationFrame callback, from what I see there are two options.

Option 1: Manually passing in the new opaque texture each frame.

const threeOpaqueTexture = new ExternalOpaqueTexture(null);
// Later on, user manually updates it.
const animate = () => {
  const glOpaqueTexture = ...;
  threeOpaqueTexture.image.source = glOpaqueTexture;
  threeOpaqueTexture.needsUpdate = true;
}

Option 2: Passing in a getter for the opaque texture in the constructor.

const getGLOpaqueTexture = () => {
  // ... interact with the XR session to get the camera feed texture
  return glOpaqueTexture;
}
const threeOpaqueTexture = new ExternalOpaqueTexture(getGLOpaqueTexture);
// Three.js internals will automatically handle re-fetching it when needed.

User API Usage

There are two ways the api might have to be used based on the limitations of an opaque texture (see above). These access patterns map to option1 and option 2 of how the underlying ExternalOpaqueTextureClass might work.

1. User must call getCameraTexture within requestAnimationFrame() of the XR session or it will error. This object is valid only for the requestAnimationFrame that it was called within and will error/warn if used otherwise.
2. User can call it at any time and receive a ExternalOpaqueTexture that will be managed by three.js, Three.js internals will call the function to get the texture inside of textures.setTexture2D. If this is called from outside of an XR sessions requestAnimationFrame it will be black.

Edited for formatting and a bit of extra information

Alternatives

I have attempted to get the camera access texture using raw webgl binds and use three.resetState to clean the three state but it doesn't work.

Additional context

Draft proposal: WebXR Raw Camera Access Module
Raw WebGL Example
Raw WebGL Example, source code
#26452

Our sales person sold a project that depends on this feature so I am going to try implement it to this spec (on r151 however). I would appreciate any advice, I know almost nothing about how three.js tracks the state internally so any help or advice would be much appreciated. If i can get it working for our project I can PR on the latest threejs version in my free time.

[connorgmeehan]

I have had a crack implementing this. I opted for the 2nd approach outlined in my initial feature request.
Here's a live example [permalink].
TinyTokyo is included as a complex scene, just to show that the GL State is not messed up.

Here's the changes, any advice or feedback on the PR would be very appreciated. Bare in mind this is based on r151 but once I have this working and can pass it off to my team I can upstream my changes to the latest three.js revision.

I am having a few issues: (never mind solved with some later commits)

• The first mesh that uses the camera feed is black, if I place another mesh they both work.
• The meshs are very susceptible to frustum culling, the texture goes black if it's even a tiny bit offscreen.
• If I exit the session and rejoin the screen is black.

One final thing, not sure on the best way of disposing the textures or if anything will need to be done to cleanup the textures seeing as they're managed by webxr.

[Mugen87]

Would you be okay to continue in #26452 and mark this issue as a duplicate? In this way, we have the discussion about the WebXR Raw Camera Access Module at one place.

[marcofugaro]

@Mugen87 that issue is more about the VideoTexture not working in that specific case, it's better to have the discussion about the Raw Camera Access Module here since @connorgmeehan has proposed some API changes.

[marcofugaro]

@connorgmeehan good job on the implementation. I believe Option 1 is more coherent to the three.js api style, the WebXRManager.getCameraTexture() function can just reuse an internal ExternalOpaqueTexture object so it's not fully exposed to the user.

[connorgmeehan]

I agree with @marcofugaro on keeping it seperate, it looks like we have 2 issues:

• The fact that having the camera-access feature enabled breaks THREE.js' internal state. When rendering a scene the texture will be bound on the first rendered element (other issue).
• How to implement a RawTexture class to handle external textures and then integrate it with the WebXRManager's and the camera-access texture. (this issue)

I just realised that my solution wasn't a complete solution to camera feed access and can have rendering artifacts depending on what's rendered in the scene (due to the other issue) so I'm going to park this issue, try to investigate and fix the other issue and then come back to this once it's done.

---

Also, yep for when I get back I am happy to re-implement with your changes @Mugen87 and using the option 1 approach.

[connorgmeehan]

Bit of a delay as I got side tracked at work. Found some previous art that we can maybe replicate. I should (hopefully) be working on this next week and PRing of the latest.
BabylonJS/Babylon.js#14527

[michaltakac]

Bit of a delay as I got side tracked at work. Found some previous art that we can maybe replicate. I should (hopefully) be working on this next week and PRing of the latest. BabylonJS/Babylon.js#14527

Any updates on this?

[mikeskydev]

I commented in the other issue as I didn't see this one. @connorgmeehan are you still interested in picking this work up, or did you make any further progress?

We would really like to use the camera access module in three.js, our use case is capturing virtual-camera aligned frames so we can do some ML in an offline process with the intrinsics, which is nicely covered in the explainer.

If there's no immediate response I'll be looking to pick this work up in the following weeks, if that's something @Mugen87 would still be interested in merging? I'm not sure how the recently announced WebGPU/WebXR compatibility will play into this, or if it's out of scope for now.