Enable post-processing in VR #20

enzofrancescaHM · 2025-01-23T18:06:01Z

This PR makes three.js (and therefore A-Frame while recompiled) compatible with Post Processing effects in VR. Without this Post Processing effects work in desktop mode but not in VR.
Refer to aframevr/aframe#3645 (comment) for previous discussions about this topic.

this modded three.js and relative A-Frame are already be tested in several examples by myself with very good results i.e. with bloom effect.

Thanks to Cody Jason Bennet for the original mods: mrdoob#26160

src/renderers/webxr/WebXRManager.js

src/renderers/WebGLRenderer.js

vincentfretin · 2025-01-24T10:23:47Z

For context, mrdoob#26160 was merged and then reverted in mrdoob#26904 see comments there about a perf issue. The pixel ratio changes were merged back again in mrdoob#26905
So the only changes remaining are what is in this PR.

dmarcos · 2025-01-24T21:13:12Z

@CodyJasonBennett would love to enable the A-Frame community to experiment with postprocessing. Thanks so much for your work.

What do you think about this PR? Any new considerations since you've worked on it? What would a cool example that performs well in VR that we can add to A-Frame to illustrate? Thanks so much

CodyJasonBennett · 2025-01-25T00:40:36Z

FWIW, the performance concerns are nonsense and trivially debunked. Also see the thread. I since added a $10k bounty on that PR mrdoob#26160 (comment) after it was clear there is infighting in Meta since Quest team's games initiative which I am supporting with react-three-fiber and some native work for Vision Pro. This is all transitory though, and eventually they will need to cooperate or lose both games and enterprise, something which they have built fierce resistance historically.

I expect the key motivation for resistance is a disgenuous attempt to use three.js to gain vendor support in specification work for oculus multiview, as their implementation has restrictions on texture use, which would line up with their concerns. For a library though, it's best to not be vendor locked, and we've used instancing tricks for similar effect prior. For users, Quest is not the only device, and it's not an option at all with enterprise and lacking QC from Meta. I've voiced these concerns in mrdoob#28968 since contributing to WebGPU/WebXR specification, as I worry vendor lock with Meta is actively bad for developers and users, post-processing being just one shut off use-case (note that it's not just post-processing affected here, but any offscreen rendering like LUTs or stencil shadow volume tricks, as this is a fix for degenerative behavior rather than a feature request).

Regarding the changes, it would indeed only need the lines that rebind to the XR layer in WebGLRenderer since I was able to backport sizing changes (which was the only functional change of the PR). I have considered implementing this as a workaround in ecosystem post-processing implementations like pmndrs/postprocessing, where a handle on the underlying XR layer render target can be gained when the session starts. Considering the impact of the changes and resistance from Meta, I thought it would be more worthwhile to upstream, as post-processing is the most requested feature since WebVR and has since regressed to include offscreen rendering entirely. The alternative would be handling XR rendering in user-land, but the design of three.js has this all or nothing with its controller APIs, so there is duplicate work to be had.

For examples, I think a good start would be a bloom effect. AMD has good literature on mobile-friendly downsamplers, and FFT bloom is very fast if physical accuracy is not paramount. pmndrs/postprocessing has a mipmap blur option in its implementation, which I have used in real projects before in VR. Your projects should be upsampling anyways for performance-constrained headsets, but that unfortunately has been shut off here as it relies on offscreen rendering. I brought this up also since three r155's removal of inline tonemapping, which requires tonemapping to be performed in user land while upsampling if no post-processing is used otherwise.

dmarcos · 2025-01-25T01:36:43Z

@CodyJasonBennett thanks for the breakdown

Any link to a post example we can borrow for A-Frame that is both cool looking and performant in VR? We can ship this in a couple of weeks in A-Frame 1.7.0

src/renderers/webxr/WebXRManager.js

deleted useless blank lines

- Deleted useless code and blank lines - refectored _getRenderTarget to getRenderTarget

Reflect the refactoring of getRenderTarget function

dmarcos · 2025-01-26T20:31:15Z

@CodyJasonBennett PR looks good to you? Thanks so much

enzofrancescaHM · 2025-01-27T08:09:09Z

@CodyJasonBennett thanks for the breakdown

Any link to a post example we can borrow for A-Frame that is both cool looking and performant in VR? We can ship this in a couple of weeks in A-Frame 1.7.0

I can arrange a small demo if wanted, problem is the library of effects, I have mine, you have yours (the one you proposed in PR in A-Frame aframevr/aframe#3645) . I don't know if you want to introduce your post-processing API in A-Frame 1.7.0, please tell me your plans and we can create a small demo with all the necessary stuff to work.

dmarcos · 2025-01-27T08:28:28Z

@enzofrancescaHM thanks. no worries about the code. just looking for an example that is good looking and performs well in a quest headset. I can port it.

enzofrancescaHM · 2025-01-27T14:09:37Z

@enzofrancescaHM thanks. no worries about the code. just looking for an example that is good looking and performs well in a quest headset. I can port it.

I have several demo ambients, this one is very low poly and has enough lights to demo bloom effect : https://incluverse.eu/examples/ppdemo.html

feel free to modify it as you want, model attributions are in the html file as comments

github-actions · 2025-01-27T20:22:21Z

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	345.58 80.56	345.69 80.58	+104 B +23 B
WebGPU	489.65 135.77	489.65 135.77	+0 B +0 B
WebGPU Nodes	489.11 135.65	489.11 135.65	+0 B +0 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	471.45 113.69	471.56 113.72	+104 B +22 B
WebGPU	559.12 151.32	559.12 151.32	+0 B +0 B
WebGPU Nodes	515.2 141.1	515.2 141.1	+0 B +0 B

dmarcos · 2025-01-27T20:28:47Z

@enzofrancescaHM Thanks. Do you have a repo with the code?

I tried with Quest 3 and perf is not great for such a simple scene. 40-70fps not stable. How does it perform without post processing?

dmarcos · 2025-01-27T20:30:05Z

@CodyJasonBennett Sorry to bother. Do you have an example of post-processing scene that performs above 72FPS consistently on Quest2/3? Thanks so much

enzofrancescaHM · 2025-01-27T20:45:59Z

@enzofrancescaHM Thanks. Do you have a repo with the code?

I tried with Quest 3 and perf is not great for such a simple scene. 40-70fps not stable. How does it perform without post processing?

this is the version with no post-processing for reference: https://incluverse.eu/examples/ppdemo_nopp.html with stats enabled. I've enabled stats also in the postprocess version for comparison.

CodyJasonBennett · 2025-01-28T07:53:16Z

I would strongly suggest https://github.com/pmndrs/postprocessing instead of three's implementation, which we had previously planned on depreciating before plans went cold. You can find a simple example with three's pass in mrdoob#26160 (comment), but try with the mipmap blur option in pmndrs/postprocessing instead.

enzofrancescaHM · 2025-01-28T16:32:26Z

I would strongly suggest https://github.com/pmndrs/postprocessing instead of three's implementation, which we had previously planned on depreciating before plans went cold. You can find a simple example with three's pass in mrdoob#26160 (comment), but try with the mipmap blur option in pmndrs/postprocessing instead.

At the moment pmndrs's postprocessing does not work in VR mode. At least I can't make it work, it works super ok in desktop mode, but not in VR mode producing black screen or white screen or in case of vignette just the vignette full screen overlay is visible. But, please, correct me if I'm wrong. an example here: https://incluverse.eu/examples/ppdemo2.html

dmarcos · 2025-01-28T17:03:34Z

@enzofrancescaHM this example that @CodyJasonBennett pointed out works in VR at 90fps (Quest3). We would need a similar example for A-Frame

https://rawcdn.githack.com/mrdoob/three.js/2bca4ee57f2c676a209d85b6b399c85475664a54/examples/webxr_vr_postprocessing.html

dmarcos · 2025-01-28T17:13:34Z

This other example in the same three.js PR also works in VR at 90fps:

https://raw.githack.com/CodyJasonBennett/three.js/test/xr-bloom/examples/webgl_postprocessing_unreal_bloom.html

I don’t like the look of the bloom though in VR. It looks like lens fogging in stereo.

Would be awesome to see a post-processing example that both looks good and performs well on-headset.

enzofrancescaHM · 2025-01-28T17:24:16Z

@enzofrancescaHM this example that @CodyJasonBennett pointed out works in VR at 90fps (Quest3). We would need a similar example for A-Frame

https://rawcdn.githack.com/mrdoob/three.js/2bca4ee57f2c676a209d85b6b399c85475664a54/examples/webxr_vr_postprocessing.html

Understood: on the fly I've prepared a very simple one: https://incluverse.eu/examples/ppquest90.html

dmarcos · 2025-01-28T17:37:57Z

@enzofrancescaHM this example that @CodyJasonBennett pointed out works in VR at 90fps (Quest3). We would need a similar example for A-Frame
https://rawcdn.githack.com/mrdoob/three.js/2bca4ee57f2c676a209d85b6b399c85475664a54/examples/webxr_vr_postprocessing.html

Understood: on the fly I've prepared a very simple one: https://incluverse.eu/examples/ppquest90.html

Thanks. Getting 50-70 fps on that example.

This one is solid 90fps:

https://raw.githack.com/CodyJasonBennett/three.js/test/xr-bloom/examples/webgl_postprocessing_unreal_bloom.html

We would also need something that is interesting to look at that makes a case for post processing in VR.

enzofrancescaHM · 2025-01-28T17:44:31Z

@enzofrancescaHM this example that @CodyJasonBennett pointed out works in VR at 90fps (Quest3). We would need a similar example for A-Frame
https://rawcdn.githack.com/mrdoob/three.js/2bca4ee57f2c676a209d85b6b399c85475664a54/examples/webxr_vr_postprocessing.html

Understood: on the fly I've prepared a very simple one: https://incluverse.eu/examples/ppquest90.html

Thanks. Getting 50-70 fps on that example.

This one is solid 90fps:

https://raw.githack.com/CodyJasonBennett/three.js/test/xr-bloom/examples/webgl_postprocessing_unreal_bloom.html

We would also need something that is interesting to look at that makes a case for post processing in VR.

if my last example is not solid 90fps there is something bad in my effects implementation. I mean... 6 cubes on a plane... something is off, for sure. Sorry for that. We should try with your own implementation, I mean the Post Process API proposed for A-Frame.

dmarcos · 2025-01-28T17:59:09Z

@enzofrancescaHM this example that @CodyJasonBennett pointed out works in VR at 90fps (Quest3). We would need a similar example for A-Frame
https://rawcdn.githack.com/mrdoob/three.js/2bca4ee57f2c676a209d85b6b399c85475664a54/examples/webxr_vr_postprocessing.html

Understood: on the fly I've prepared a very simple one: https://incluverse.eu/examples/ppquest90.html

Thanks. Getting 50-70 fps on that example.
This one is solid 90fps:
https://raw.githack.com/CodyJasonBennett/three.js/test/xr-bloom/examples/webgl_postprocessing_unreal_bloom.html
We would also need something that is interesting to look at that makes a case for post processing in VR.

if my last example is not solid 90fps there is something bad in my effects implementation. I mean... 6 cubes on a plane... something is off, for sure. Sorry for that. We should try with your own implementation, I mean the Post Process API proposed for A-Frame.

I would not go for a post processing API right of the bat. A starting point would be merging this PR and an ad-hoc simple example in A-Frame to show how you can quickly integrate and start playing with it. Does it sound good?

dmarcos · 2025-01-28T18:00:01Z

Also atm post-processing is not top priority for me so need others to step in.

enzofrancescaHM · 2025-01-28T18:53:51Z

Sounds good to me, yes and thank you! I can offer my time to build examples, but not to go in-depth in post-processing effects code, it is too much complex for the time I can dedicate.
And of course it would be amazing if others step in to help. Maybe I can throw some stone into the water on the discord channel :) :) :)

CodyJasonBennett · 2025-01-28T19:17:22Z

I'm afraid I won't have access to a headset until the weekend, but I'd be happy to contribute an example.

dmarcos · 2025-01-28T19:31:39Z

@CodyJasonBennett thanks so much. will be cool to have a cool looking at performant example to get people excited. Is it this PR good to merge as is?

@enzofrancescaHM Can you submit an integration example to https://github.com/aframevr/aframe/tree/master/examples? maybe a bloom component as simple as you can make it. We can swap the effect later with @CodyJasonBennett example.

Thanks everyone

enzofrancescaHM · 2025-01-29T10:01:14Z

@enzofrancescaHM Can you submit an integration example to https://github.com/aframevr/aframe/tree/master/examples? maybe a bloom component as simple as you can make it. We can swap the effect later with @CodyJasonBennett example.

Thanks everyone

Yes, I'm preparing a simple example with just bloom and I am going to submit to a-frame. Just for curiosity, what method do you use for getting FPS on Quest's browser? So I can align with you with my tests.

dmarcos · 2025-01-29T13:09:48Z

@enzofrancescaHM Can you submit an integration example to https://github.com/aframevr/aframe/tree/master/examples? maybe a bloom component as simple as you can make it. We can swap the effect later with @CodyJasonBennett example.
Thanks everyone

Yes, I'm preparing a simple example with just bloom and I am going to submit to a-frame. Just for curiosity, what method do you use for getting FPS on Quest's browser? So I can align with you with my tests.

Thanks! I use the OVR Metrics tool HUD:

https://www.meta.com/experiences/ovr-metrics-tool/2372625889463779

enzofrancescaHM · 2025-01-29T14:24:31Z

@enzofrancescaHM Can you submit an integration example to https://github.com/aframevr/aframe/tree/master/examples? maybe a bloom component as simple as you can make it. We can swap the effect later with @CodyJasonBennett example.
Thanks everyone

Yes, I'm preparing a simple example with just bloom and I am going to submit to a-frame. Just for curiosity, what method do you use for getting FPS on Quest's browser? So I can align with you with my tests.

Thanks! I use the OVR Metrics tool HUD:

https://www.meta.com/experiences/ovr-metrics-tool/2372625889463779

ok, something is odd: I've installed OVR Metrics Tool as well and I can't see any difference in FPS between same scene with or without post-processing:

https://incluverse.eu/examples/ppquest90n.html (version NO post-processing)
https://incluverse.eu/examples/ppquest90.html (version WITH post-processing)

dmarcos · 2025-01-29T15:06:14Z

Start with a 90fps baseline like the hello world

https://aframe.io/aframe/examples/boilerplate/hello-world/

add stuff progressively until perf degrades

enzofrancescaHM · 2025-01-29T16:39:32Z

Start with a 90fps baseline like the hello world

https://aframe.io/aframe/examples/boilerplate/hello-world/

add stuff progressively until perf degrades

yeah, thanks for the advice... 35 years of development and still I fall into beginner traps. Sorry for that, now this example should work 90 fps steady with post processing: https://incluverse.eu/examples/hello.html

dmarcos · 2025-01-29T17:53:05Z

@enzofrancescaHM cool! thanks! can you open a PR in the A-Frame repo with the example? https://github.com/aframevr/aframe

We can later replace the scene later with @CodyJasonBennett effect

enzofrancescaHM · 2025-01-30T09:57:11Z

@enzofrancescaHM cool! thanks! can you open a PR in the A-Frame repo with the example? https://github.com/aframevr/aframe

We can later replace the scene later with @CodyJasonBennett effect

ok, PR with example done in A-Frame Repo.

dmarcos · 2025-01-31T08:02:19Z

Thanks so much for sticking with it

* Update WebXRManager.js * Update WebGLRenderer.js * Update WebGLRenderer.js deleted useless blank lines * Update WebXRManager.js - Deleted useless code and blank lines - refectored _getRenderTarget to getRenderTarget * Update WebGLRenderer.js Reflect the refactoring of getRenderTarget function

This reverts commit d10c27a.

* Update WebXRManager.js * Update WebGLRenderer.js * Update WebGLRenderer.js deleted useless blank lines * Update WebXRManager.js - Deleted useless code and blank lines - refectored _getRenderTarget to getRenderTarget * Update WebGLRenderer.js Reflect the refactoring of getRenderTarget function

* Update WebXRManager.js * Update WebGLRenderer.js * Update WebGLRenderer.js deleted useless blank lines * Update WebXRManager.js - Deleted useless code and blank lines - refectored _getRenderTarget to getRenderTarget * Update WebGLRenderer.js Reflect the refactoring of getRenderTarget function Improve compatibility of Post-Processing with VR (#23) * Update EffectComposer.js for better compatibility with VR * Update Pass.js for better compatibility with VR

enzofrancescaHM added 2 commits January 23, 2025 18:49

Update WebXRManager.js

e49ce01

Update WebGLRenderer.js

02e260d

vincentfretin suggested changes Jan 24, 2025

View reviewed changes

src/renderers/webxr/WebXRManager.js Outdated Show resolved Hide resolved

src/renderers/webxr/WebXRManager.js Outdated Show resolved Hide resolved

src/renderers/webxr/WebXRManager.js Outdated Show resolved Hide resolved

src/renderers/WebGLRenderer.js Outdated Show resolved Hide resolved

dmarcos force-pushed the dev branch 2 times, most recently from 1d922fd to b10b9ad Compare January 24, 2025 21:08

dmarcos reviewed Jan 25, 2025

View reviewed changes

src/renderers/webxr/WebXRManager.js Outdated Show resolved Hide resolved

enzofrancescaHM added 3 commits January 26, 2025 11:12

Update WebGLRenderer.js

0378b1a

deleted useless blank lines

Update WebXRManager.js

75c550c

- Deleted useless code and blank lines - refectored _getRenderTarget to getRenderTarget

Update WebGLRenderer.js

6ed1447

Reflect the refactoring of getRenderTarget function

dmarcos changed the title ~~ThreeJS compatibility with PostProcessing in VR~~ Enable post-processing in VR Jan 27, 2025

enzofrancescaHM mentioned this pull request Jan 30, 2025

Post-Process example aframevr/aframe#5648

Merged

dmarcos merged commit 75f08d9 into supermedium:dev Jan 31, 2025
9 of 11 checks passed

vincentfretin added a commit to vincentfretin/three.js that referenced this pull request Jan 31, 2025

Revert "Enable post-processing in VR (supermedium#20)"

894dd3b

This reverts commit d10c27a.

enzofrancescaHM mentioned this pull request Feb 1, 2025

Improve compatibility of Post-Processing with VR #23

Merged

mrxz mentioned this pull request Feb 26, 2025

Bug on exitVR (quest 2) aframevr/aframe#5678

Closed

Enable post-processing in VR #20

Enable post-processing in VR #20

Uh oh!

Conversation

enzofrancescaHM commented Jan 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vincentfretin commented Jan 24, 2025

Uh oh!

dmarcos commented Jan 24, 2025

Uh oh!

CodyJasonBennett commented Jan 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dmarcos commented Jan 25, 2025

Uh oh!

Uh oh!

dmarcos commented Jan 26, 2025

Uh oh!

enzofrancescaHM commented Jan 27, 2025

Uh oh!

dmarcos commented Jan 27, 2025

Uh oh!

enzofrancescaHM commented Jan 27, 2025

Uh oh!

github-actions bot commented Jan 27, 2025

📦 Bundle size

🌳 Bundle size after tree-shaking

Uh oh!

dmarcos commented Jan 27, 2025

Uh oh!

dmarcos commented Jan 27, 2025

Uh oh!

enzofrancescaHM commented Jan 27, 2025

Uh oh!

CodyJasonBennett commented Jan 28, 2025

Uh oh!

enzofrancescaHM commented Jan 28, 2025

Uh oh!

dmarcos commented Jan 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dmarcos commented Jan 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

enzofrancescaHM commented Jan 28, 2025

Uh oh!

dmarcos commented Jan 28, 2025

Uh oh!

enzofrancescaHM commented Jan 28, 2025

Uh oh!

dmarcos commented Jan 28, 2025

Uh oh!

dmarcos commented Jan 28, 2025

Uh oh!

enzofrancescaHM commented Jan 28, 2025

Uh oh!

CodyJasonBennett commented Jan 28, 2025

Uh oh!

dmarcos commented Jan 28, 2025

Uh oh!

enzofrancescaHM commented Jan 29, 2025

Uh oh!

dmarcos commented Jan 29, 2025

Uh oh!

enzofrancescaHM commented Jan 29, 2025

Uh oh!

dmarcos commented Jan 29, 2025

Uh oh!

enzofrancescaHM commented Jan 29, 2025

Uh oh!

dmarcos commented Jan 29, 2025

Uh oh!

enzofrancescaHM commented Jan 30, 2025

Uh oh!

dmarcos commented Jan 31, 2025

Uh oh!

CodyJasonBennett commented Jan 25, 2025 •

edited

Loading

dmarcos commented Jan 28, 2025 •

edited

Loading

dmarcos commented Jan 28, 2025 •

edited

Loading