WebXR: Frustum culling inconsistency between renderers

[harryhjsh]

Description

Frustum culling for ArrayCameras was changed in #30830 for Renderer: a FrustumArray now culls for each camera independently.

For XR sessions with two views, XRManager and WebXRManager both use setProjectionFromUnion to compute a combined projection matrix for the XR ArrayCamera.

In WebGLRenderer, this is used for culling in the same way as any other camera.

In Renderer, the new FrustumArray takes over and each camera from the XR ArrayCamera is culled separately.

I think the inconsistency could be confusing, and the result of setProjectionFromUnion is now never used in the new renderer.

Solution

• Backport the FrustumArray culling into WebXRManager to fix the consistency issue.

• If we want to keep the setProjectionFromUnion behaviour usable for frustum culling, we need to provide some way for (Web)XRManager to opt-out of FrustumArray culling when it only has two views.

This being opt-out rather than opt-in for XR would also fix the culling behaviour for XR devices with > 2 views, which currently don't cull properly in WebGLRenderer (but now do in Renderer, albeit per-view) - see #19387. (Full disclosure that I am working on something downstream which could make use of this).

See also: #15054 (comment) - we could also close that issue if we think that per-camera culling serves the same purpose as combining the frustra for now.

Alternatives

I think the consistency between the renderers is important, and the array camera behaviour is useful generally.

If there isn't appetite for the specific opt-out for the XRManagers, we could consider removing the setProjectionFromUnion path and just using the FrustumArray culling. I'm not sure what kind of performance difference it actually makes, given that - even without it - only one culling pass would probably be performed a not-insignificant amount of the time. Removing it could end up being cleaner.

N.B. that it seems like there's at least some userland usage of the combined projection matrix: #21002

Additional context

Some Background:

• Expose combined frustum from both eyes immersive-web/performance-improvements#4
• Expose a hidden area mesh or a hidden area depth write function immersive-web/performance-improvements#1 (comment)

[harryhjsh]

PR open (#31677), more thoughts appreciated (and happy to split it up if that's helpful).

[FrostKiwi]

Just stumbled upon Quest 3 showing objects correctly in immersive-ar but the Pixel 9 hiding some Objects in WebXR, until I disable frustum culling for those. It's a bit strange, I gotta essentially fully disable frustum culling for my scene to render correctly across all platforms.

I tested out this PR and it did not solve the problem. Any idea how to debug this properly? Console log WebXR Camera positions and object bounding boxes or something?

[FrostKiwi]

Ohh, figured it out!
With Pixel 9 WebXR, ThreeJS thinks the camera is still at the origin! Whilst Quest 3 WebXR, the camera.position updates properly.
It turned out that https://github.com/yomotsu/camera-controls and WebXR don't mix on Android WebXR. What a bizzare interaction.

[harryhjsh]

Obviously you've worked this out now, but there is a chance your problem could've been solved with the related PR + opting in to per-camera culling using renderer.xr.getCamera().perCameraCulling = true. I haven't looked at all at the internals of that controls library, and whatever your Android XR implementation is passing to three.js is probably having an impact, but even with this PR you'd be using the combined frustum discussed above by default.

If your scene was otherwise rendering as expected but the controls library was doing something weird to your camera/the XR camera's position, performing the culling per-camera might have solved whatever it was.

Worth mentioning too that if per-camera culling resulted in your scene rendering as expected, that would've been the default behaviour with the current release's WebGPURenderer (which currently does per-camera culling for every ArrayCamera it's passed by default) -- but not the current release's WebGLRenderer (which can only ever use the combined frustum).