WebXRManager: Add Raw Camera Access module.

[mikeskydev]

Closes #26452, Closes #26891

Description

This PR implements support for the WebXR Raw Camera Access module, and provides an example where the camera texture is placed on a rotating cube.

I have addressed the issue in #26452 by unbinding texture state if camera access is enabled, as it appears the module will automatically bind the raw camera textures at the beginning of the frame, which was affecting three.js's internal state tracking.

As this was tested on Chrome for android, this may be an implementation bug in Blink that has gone unnoticed, as their test case always rebinds textures.

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	338.39 78.91	338.82 79.08	+428 B +163 B
WebGPU	566.16 156.5	566.16 156.5	+0 B +0 B
WebGPU Nodes	564.77 156.26	564.77 156.26	+0 B +0 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	469.82 113.62	470.25 113.78	+428 B +155 B
WebGPU	637.38 172.47	637.38 172.47	+0 B +0 B
WebGPU Nodes	592.02 161.71	592.02 161.71	+0 B +0 B

[mikeskydev]

Any chance this makes r179? My actual use case is further downstream... :)

[Mugen87]

I've removed the opaque property from RawTexture because the renderer does not evaluate the flag anymore. Also committed some JSDoc improvements.

Do you mind temporarily updating the builds so I can test the PR with a Quest 2 over https://rawcdn.githack.com/?

[mikeskydev]

Thanks! I have a built version for my own purposes on https://github.com/mikeskydev/three.js/tree/cam-access-build , but be aware camera-access isn't available on Quest yet. I've been working for mobile, and it should be supported on a default Android Chrome with no extra flags

[Mugen87]

Thanks for the heads-up. I can still test if the depth sensing demos work so we have no regression on that side.

https://rawcdn.githack.com/mikeskydev/three.js/98bc573b5d9ea8280116d100497ecadf08260139/examples/index.html

[Mugen87]

There seems to be no change in behavior 👍 .

However, it seems like depth sensing is currently broken in production. When I understand webxr_xr_dragging_custom_depth correctly, the meshes should be occluded by objects in the real world (see https://x.com/rcabanier/status/1748050448270135425). However, I don't get objects occluded at least on my Quest 2. They are always rendered in front of everything.

There are no WebGL/WebXR warnings in the browser console when using remote debugging.

/ping @cabanier I'm afraid a regression has sneaked in. Could you please have a look?

[mikeskydev]

I tried the depth example on my phone (which doesn't support depth) and it seems to silently fail as it's an optional feature, so it may also be depth sensing was turned off for quest browser at some point?

[Mugen87]

Oh wait, it seems depth sensing only works with a Quest 3 since the Quest 2 does not have the required depth sensor.

@cabanier Sorry for the ping 😇 .

[cabanier]

Oh wait, it seems depth sensing only works with a Quest 3 since the Quest 2 does not have the required depth sensor.

Depth sensing is no longer working with these changes. I'll investigate why.

[cabanier]

Oh wait, it seems depth sensing only works with a Quest 3 since the Quest 2 does not have the required depth sensor.

Depth sensing is no longer working with these changes. I'll investigate why.

If I revert 6692d4d (Flip bool), depth sensing works again

[Mugen87]

The texture retrieved via getDepthInformation() just needs to be bound for usage, right? There is no need for an explicit texture data upload.

We've introduced RawTexture to have a dedicated code path for this use case and avoid code sections like:

const texProps = renderer.properties.get( texture );
texProps.__webglTexture = depthData.texture;

If the depth sensing texture is wrapped into a RawTexture instance, the outcome should be the same. Especially since nowhere in WebXRDepthSensing texture.needsUpdate is set to true. I wonder what me miss 🤔 .

[mikeskydev]

Technically opaque textures should be bound every frame as they go invalid, In camera access it doesn't work unless I rebind in the loop.

Any chance that depth was also working by a fluke and is also affected by the unbind?

[cabanier]

Technically opaque textures should be bound every frame as they go invalid, In camera access it doesn't work unless I rebind in the loop.

Any chance that depth was also working by a fluke and is also affected by the unbind?

Yes, you need to rebind the textures every frame. This is because we play tricks under the hood and replace the underlying system texture.

[Mugen87]

@mikeskydev Thanks for working on camera access and implementing the RawTexture solution. It's great to see that also the depth sensing code path benefits from the new texture type.

[mikeskydev]

Agreed, thanks for pushing for the cleaner solution, helping get the PR over the line and for your time, it's really appreciated!

[mrdoob]

Maybe ExternalTexture could be a better name?
Are there other uses for it? If not, may be WebXRCameraTexture?

[mikeskydev]

We also use it for webxr depth textures, but there was some debate about naming in the linked issues and RawTexture was settled on. I could see it being useful for anything that uses raw texture handles, possibly protected content?