Add preliminary support for WebXR Layers #20696

cabanier · 2020-11-17T18:59:03Z

Description

WebXR Layers is an upcoming standard that allows more efficient projections.
This change introduces an initial implementation which lets three.js draw in a projection layer. It also has an example how one can merge it with Quad and Equirect media layers.

This contribution is funded by Oculus.

examples/webxr_layers.html

mrdoob · 2020-11-21T16:19:16Z

Hmm, this one requires a deeper review...

examples/files.json

mrdoob · 2020-12-21T22:38:40Z

This is to be able to display an image or a video in VR in a more performant and crisper way, correct?

cabanier · 2020-12-21T23:36:42Z

This is to be able to display an image or a video in VR in a more performant and crisper way, correct?

This PR makes it possible to use such constructs. Currently, it enables layer support and draws content to a projection layer instead of a WebGL layer. Projection layers are slightly more efficient and also support texture arrays which could be enabled later.
Once an experience switches to layers, it can use media layers to easily show crisp video or gl layers to draw crisp content.

mrdoob · 2020-12-22T00:11:07Z

Do these layers use the same depth buffer? Or do they render on top of gl?

cabanier · 2020-12-22T00:19:25Z

Do these layers use the same depth buffer? Or do they render on top of gl?

Each layer has its own depth buffer and it's up to the user agent if they want to use it for reprojection/timewarp.
In case of the Oculus, we discard the buffer (= we don't use it in the VR Compositor)

mrdoob · 2020-12-22T00:34:02Z

What I'm trying to understand is whether the video will mix with the 3d scene or not. For example: A video layer surrounded by gl spheres.

But by your response I deduce that the gl scene will be rendered first and the video/layer will be rendered on top ignoring the depth buffer of the gl scene.

Correct?

cabanier · 2020-12-22T00:38:09Z

That is correct. Layers are rendered on top of each other in the order of the layers array.
There is not depth sorting that happens in the system compositor.

Typically, the controller and a simplified scene is rendered on top and then there are quad/cylinder layers for video or custom controls.
The background is then filled in with a cube or equirect layer.

mrdoob · 2020-12-22T13:16:56Z

src/renderers/webxr/WebXRManager.js

-				stencil: attributes.stencil,
-				framebufferScaleFactor: framebufferScaleFactor
-			};
+			if ( session.renderState.layers === undefined ) {


Are we going to have to maintain two code paths from now on?
Or is this transitional and we'll able to just use the layers code path in the future?

I know Microsoft is going to start on layers and Google also said that they're interested because it gives them access to texture arrays and in the near future WebGPU.
Once they land support, we can remove the dual path.

mrdoob · 2020-12-22T13:29:56Z

So many questions...

Typically, the controller and a simplified scene is rendered on top

The controller? The 3d model of the controllers?
What do you mean with "simplified" scene?
And, rendered on top of what? On top of the layers?

and then there are quad/cylinder layers for video or custom controls.

Okay, I understand this.

The background is then filled in with a cube or equirect layer.

So, ideally we should be rendering scene.background in VR like this, right?

Also.. sounds like cube or equirect layer do not render on top of everything? They render under everything, correct?

If so, what happens with transparent objects. I've experimented in the past with rendering the background as the last step in order to save on fillrate, but the idea failed when the scene had transparent objects.

cabanier · 2020-12-22T14:32:14Z

So many questions...

Typically, the controller and a simplified scene is rendered on top

The controller? The 3d model of the controllers?

Yes. For instance, if you take the oculus shell, you can see that the controllers, the ray and the hands are rendered on top of everything else. This is done in a projection layer.

What do you mean with "simplified" scene?

You can see this in a number of games. For instance, if you look at "the climb" or "falcon age", the characters and nearby bits of the scene are drawn to a projection layers. The background is typically a cubemap that is static or with some simple canned animations.
Because the background can be drawn at a lower framerate and in a simpler way, the GPU and CPU have more headroom to spend on the foreground content.

And, rendered on top of what? On top of the layers?

Yes

and then there are quad/cylinder layers for video or custom controls.

Okay, I understand this.

The background is then filled in with a cube or equirect layer.

So, ideally we should be rendering scene.background in VR like this, right?

Can the background be complex? (ie like the background in the oculus shell) If so, yes.
Also because the system compositor knows that it is drawing content from an equirect or cubemap, it can render the content faster, sharper and with less distortion.

Also.. sounds like cube or equirect layer do not render on top of everything? They render under everything, correct?

It depends on their order in the layers array but typically, they render under everything else.

If so, what happens with transparent objects. I've experimented in the past with rendering the background as the last step in order to save on fillrate, but the idea failed when the scene had transparent objects.

That depends on the blendtexturesourcealpha attribute. By default layers will alpha blend with each other so transparent content will be handled correctly.

mrdoob · 2020-12-22T18:11:03Z

examples/webxr_vr_layers.html

+				if ( session && session.renderState.layers !== undefined && session.hasMediaLayer === undefined && video.readyState >= 2 && video2.readyState >= 2) {
+					session.hasMediaLayer = true;
+					session.requestReferenceSpace('local').then((refSpace) => {
+						const mediaBinding = new XRMediaBinding(session);
+						const equirectLayer = mediaBinding.createEquirectLayer(video, {space: refSpace, layout: "mono"});
+						const quadLayer = mediaBinding.createQuadLayer(video2, {space: refSpace, layout: "stereo-left-right"});
+						quadLayer.transform = new XRRigidTransform({x: 1.5, y: 1.0, z: -2.0});
+						session.updateRenderState( { layers: [ equirectLayer, quadLayer, session.renderState.layers[0] ] } );
+						video.play();
+						video2.play();
+					});
+				}


I think we could create a XRMediaLayer class that extends Object3D so we can add it to the scene like any other object. And then let WebXRManager do all this binging stuff when it sees a XRMediaLayer.

What do you think of that design?

That sounds reasonable.
How would you handle WebGL Layers?

How would you handle WebGL Layers?

Can you elaborate?

WebGL layers give you another texture to render to.
Just like how a projection layer gives you a stereo texture, other layer types can give you one.
Can three.js draw into multiple textures at a time? non-projection layers are also special in that they don't have to be drawn at the refresh rate.

No, it can't draw into multiple textures yet.

I was not aware of WebGL layers... Seems like I'll have to do a proper read at the spec. I'm too confused right now 😅

We can tackle those later :-)
I believe this project updated three.js so it could target multiple layers at the same time: https://spellew.github.io/webxr-samples/media-layers.html

Is there a talk that explains Layers? Or is https://immersive-web.github.io/layers/ the only resource?

That is the only resource for the WebXR part.
For general information on layers, you can read:

https://developer.oculus.com/documentation/unity/unity-ovroverlay/?locale=en_US

https://developer.oculus.com/documentation/unreal/unreal-overlay/?locale=en_US

https://www.khronos.org/registry/OpenXR/specs/1.0/html/xrspec.html#rendering

@dmarcos also hooked them into aframe: https://twitter.com/dmarcos/status/1322309762562760704

cabanier · 2020-12-28T23:12:04Z

@mrdoob did you have a chance to think about how to best integrate layers?

cabanier · 2021-02-22T22:59:33Z

@mrdoob We're looking into creating some examples with WebXR Layers and three.js.
Everything is working but we're wondering how we can render to separate surfaces. Is there a convenient way to do that?

mrdoob · 2021-02-22T23:55:54Z

Everything is working but we're wondering how we can render to separate surfaces.

What do you mean with surfaces?

cabanier · 2021-02-23T07:49:22Z

Each WebXR layer is pointing to an opaque texture.
This means that we'll need multiple renderers that can draw to each one of them and using the same GL context. Does that require changes in three.js?

mrdoob · 2021-02-23T12:05:23Z

Sounds like something similar to multi render targets?

cabanier · 2021-02-23T14:10:14Z

Yes, is that already available?

mrdoob · 2021-03-02T15:22:38Z

Not yet. I was waiting for WebGL2 landing on iOS so we don't have multiple code paths.

cabanier · 2021-03-02T17:36:50Z

Is it already in progress on a branch or a PR?

Mugen87 · 2021-03-02T18:20:59Z

Yes, see #16390.

Mugen87 · 2021-03-02T18:23:08Z

Sidenote: #20135 is required for the MRT PR so it's good if we manage to merge it this dev cycle.

mrdoob · 2021-06-26T09:04:55Z

@cabanier We have MRT support now. Would you like to do any changes to this PR?

cabanier · 2021-06-26T18:35:44Z

@cabanier We have MRT support now. Would you like to do any changes to this PR?

I think this PR is a good start. We can add better support for layers in later patches (ie render to more than 1 layer)

mrdoob · 2021-06-28T15:35:28Z

Sounds good!

Any chance you can resolve the conflicts?

cabanier · 2021-06-28T16:58:43Z

Sounds good!

Any chance you can resolve the conflicts?

Will do!

cabanier · 2021-06-28T23:07:21Z

Sounds good!

Any chance you can resolve the conflicts?

I resolved the conflicts and ran some tests.
One of the test files that used to work is now broken: https://threejs.org/examples/?q=sandbox#webxr_vr_sandbox
It seems that there is now a problem when switching render targets. Do you know someone who can help me?

cabanier · 2021-06-29T14:30:53Z

#22060 has the latest changes.
https://threejs.org/examples/?q=sandbox#webxr_vr_sandbox is broken because of a bug in the browser. That will be fixed in the next release.

mrdoob reviewed Nov 21, 2020

View reviewed changes

examples/webxr_layers.html Outdated Show resolved Hide resolved

mrdoob added this to the r124 milestone Nov 21, 2020

cabanier added 2 commits November 24, 2020 04:11

Add preliminary support for WebXR Layers

485ea58

WebXR Layers: create the correct type of texture/depth textures

a2f772a

cabanier mentioned this pull request Dec 8, 2020

Mirror Example broken on Standalone Oculus Quest #19759

Closed

11 tasks

mrdoob reviewed Dec 21, 2020

View reviewed changes

examples/files.json Outdated Show resolved Hide resolved

WebXR Layers: rename sample

d3ceae1

mrdoob reviewed Dec 22, 2020

View reviewed changes

mrdoob modified the milestones: r124, r125 Dec 22, 2020

mrdoob mentioned this pull request Dec 27, 2020

Added "controllermove" event to WebXRController #20790

Merged

arodic mentioned this pull request Dec 27, 2020

WebXRManager: Added setReferenceSpace() #20949

Merged

mrdoob modified the milestones: r125, r126 Jan 27, 2021

mrdoob modified the milestones: r126, r127 Feb 22, 2021

mrdoob modified the milestones: r127, r128 Mar 30, 2021

mrdoob modified the milestones: r128, r129 Apr 23, 2021

mrdoob modified the milestones: r129, r130 May 27, 2021

cabanier closed this Jun 29, 2021

mrdoob removed this from the r130 milestone Jun 29, 2021

Uh oh!

Add preliminary support for WebXR Layers #20696

Add preliminary support for WebXR Layers #20696

Uh oh!

Conversation

cabanier commented Nov 17, 2020

Uh oh!

Uh oh!

mrdoob commented Nov 21, 2020

Uh oh!

Uh oh!

mrdoob commented Dec 21, 2020

Uh oh!

cabanier commented Dec 21, 2020

Uh oh!

mrdoob commented Dec 22, 2020

Uh oh!

cabanier commented Dec 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrdoob commented Dec 22, 2020

Uh oh!

cabanier commented Dec 22, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrdoob commented Dec 22, 2020

Uh oh!

cabanier commented Dec 22, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cabanier commented Dec 28, 2020

Uh oh!

cabanier commented Feb 22, 2021

Uh oh!

mrdoob commented Feb 22, 2021

Uh oh!

cabanier commented Feb 23, 2021

Uh oh!

mrdoob commented Feb 23, 2021

Uh oh!

cabanier commented Feb 23, 2021

Uh oh!

mrdoob commented Mar 2, 2021

Uh oh!

cabanier commented Mar 2, 2021

Uh oh!

Mugen87 commented Mar 2, 2021

Uh oh!

Mugen87 commented Mar 2, 2021

Uh oh!

mrdoob commented Jun 26, 2021

Uh oh!

cabanier commented Jun 26, 2021

Uh oh!

mrdoob commented Jun 28, 2021

Uh oh!

cabanier commented Jun 28, 2021

Uh oh!

cabanier commented Jun 28, 2021

Uh oh!

cabanier commented Dec 22, 2020 •

edited

Loading