WebGLRenderer: Add outputBufferType and setEffects()

[mrdoob]

Related issue: #15840 #18322 #18846 #32433 #32449

Description

Adds internal HDR rendering and simplified post-processing API.

Changes

• outputBufferType option for HalfFloatType rendering
• setEffects() method for post-processing without EffectComposer
• isRenderPass flag for TAARenderPass/SSAARenderPass support

const renderer = new WebGLRenderer({ outputBufferType: HalfFloatType });
renderer.setEffects( [ taaPass, bloomPass ] );
renderer.render( scene, camera );

Before	After

The watch glass HDR is fixed but the text becomes less sharp. I still don't understand why...

Before	After

For testing:
https://raw.githack.com/mrdoob/three.js/733f7abc2378874b0c386010b675b7a43f5c6aa5/examples/index.html

[github-actions]

📦 Bundle size

Full ESM build, minified and gzipped.

	Before	After	Diff
WebGL	350.93 83.27	355.12 84.44	+4.19 kB +1.17 kB
WebGPU	616.43 171.09	616.43 171.09	+0 B +0 B
WebGPU Nodes	615.03 170.83	615.03 170.83	+0 B +0 B

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

	Before	After	Diff
WebGL	483.23 118.17	487.46 119.37	+4.23 kB +1.2 kB
WebGPU	687.34 186.69	687.34 186.69	+0 B +0 B
WebGPU Nodes	637.18 173.9	637.18 173.9	+0 B +0 B

[mrdoob]

One downside of this approach is that we can no longer render specific materials (like MeshNormalMaterial) without tonemapping.

Edit:
Unless we "detonemap" them... 👀

[Mugen87]

To clarify, this PR would align WebGLRenderer to WebGPURenderer and remove the inline tone mapping and color space conversion.

I suspect this change will produce quite some rejection in the community since there are still serious concerns about the performance of WebGPURenderer on mobile and XR devices. The double pass approach is quite noticeable performance-wise and we haven't finally dismissed a inline approach for performance reasons in WebGPURenderer.

I suggest you make this optional for WebGLRenderer and keep the inline approach as a default. There are definitely users that don't want such a system in WebGLRenderer right now.

/cc @donmccurdy

[mrdoob]

This is only a test PR to see how the renders are supposed to look like.
And also to see how the code would look like.

[Mugen87]

BTW: An optional setup with WebGLRenderer is via EffectComposer. A setup of a render pass + output psass should produce similar results.

[Mugen87]

This is only a test PR to see how the renders are supposed to look like. And also to see how the code would look like.

Ah I understand. I'm all in for aligning the renderers. But given the deep character of this change, we should make clear to retain the inline approach in WebGLRenderer for backwards-compatibility reasons. And even so if the performance differences between one and two passes turn out smaller than expected.

Here is a latest issue about that topic: #32273

[mrdoob]

Refactored the PR:

• Added colorBufferType to WebGLRenderer (UnsignedByteType by default)
• Also added .setEffects() which tries to simplify the EffectComposer UX (experimental, I'm going to see how much code its needed to get bloom in VR, may revert this)

@gkjohnson Changing colorBufferType to solve the clamped transparency issues feels much more easier to understand to me than messing with premultiplied alpha. Both options have their downsides but this one seems better as it also aligns with WebGPURenderer.

[mrdoob]

@sunag I think outputBufferType would be a better name than colorBufferType. Did you consider that name at the time?

[sunag]

@sunag I think outputBufferType would be a better name than colorBufferType. Did you consider that name at the time?

I agree. I didn't consider it in the past.
I will rename it.

@Mugen87 Is this okay for you too?

[donmccurdy]

Having both outputType and outputBufferType names are going to be hard to differentiate. Perhaps:

• (a) outputType could be renamed to canvasBufferType, drawingBufferType, or contextType. I believe the nearest equivalent Web API terms are "context format" in WebGPU (in spec) and "drawing buffer format" in WebGL (draft only?).
• (b) colorBufferType could be renamed to compositingBufferType, frameBufferType, or perhaps backBufferType. This is the step before output to canvas, and outputColorSpace describes the color space of the canvas, not the frame buffer, so I'm very hesitant to use the word "output" for the internal frame buffer. Keeping it named "colorBufferType" is also OK with me.

[sunag]

I think we could remove term buffer to define screen output associated with the hardware?
It seems to me that it would fit better associated with textures.

I was thinking about the table below.

Name	Description
.outputType	Screen/Output hardware target bits precision and range
.outputBufferType	Working bits precision and range

It also occurred to me to use workingBufferType instead of outputBufferType, since we also use the term in workingColorSpace.

[donmccurdy]

Thanks! Now that you frame it that way it makes more sense to me. I think either of those suggestions could work, yes.

[mrdoob]

• Also added .setEffects() which tries to simplify the EffectComposer UX (experimental, I'm going to see how much code its needed to get bloom in VR, may revert this)

I managed to get bloom working inside VR 😱

I don't have a headset to try it with at the moment...

Can anyone try it on Quest, GalaxyXR, Vision Pro, etc?
https://raw.githack.com/mrdoob/three.js/733f7abc2378874b0c386010b675b7a43f5c6aa5/examples/index.html?q=webxr#webxr_xr_controls_transform

[mrdoob]

Ops, looks like webgl_watch broke. I'll fix it later.

[mrdoob]

Passthrough doesn't work either.
Probably the good old unreal bloom alpha issue...

[jjmhalew]

Can anyone try it on Quest, GalaxyXR, Vision Pro, etc?
https://raw.githack.com/mrdoob/three.js/webgl-internal-msaa-rt/examples/index.html?q=webxr#webxr_xr_controls_transform

Quest 2

VR.test.mp4

[ruofeidu]

Thank you mrdoob!

Forgot to tell you that we had postprocessing with stereo depth in WebXR here: https://xrblocks.github.io/docs/samples/DepthMap/

Glad that you fixed threejs Effect!!

[donmccurdy]

Name	Description
.outputType	Screen/Output hardware target bits precision and range
.outputBufferType	Working bits precision and range

I think I need to clarify, this definition was OK with me assuming that .outputType described the target canvas format, i.e. the thing we pass into context.configure({format: '...'}). I took "working bits precision and range" to refer to internal buffers in the working color space, such as the frame buffer used as input to tone mapping.

Neither of these describes the screen/hardware capabilities, we can't do that within a browser environment, we only configure the canvas and declare its content characteristics (e.g. color space).

[mrdoob]

@ruofeidu

Forgot to tell you that we had postprocessing with stereo depth in WebXR here: https://xrblocks.github.io/docs/samples/DepthMap/

Oh, I currently con't have a headset and this is what I'm getting with Meta's emulator:

[donmccurdy]

Instead of .outputType, how about calling it [ canvasFormat | canvasType | canvasFormatType ]?

outputType's original behavior was to manage context.configure({format: '???'}), for which I agree these would be good names. The proposal @sunag above would change that, so outputType declares the contents of the canvas to be imagery in an HDR range, expressing creative intent and not buffer/texture formats, while outputBufferType configures the canvas type / format with context.configure({...}) as outputType did before. Certainly those two concepts are distinct.

I'm not yet comfortable with the outputType/outputBufferType division of responsibilities, but we're still discussing it at #32508. I'm inclined to say workingBufferType or colorBufferType are better names for the internal framebuffer, with outputType (renamed canvasType if you prefer) configuring the canvas context as before. Our naming of "output", "canvas", and "working" roles should be consistent across -Type and -ColorSpace properties. And then TBD on property naming for declaring that the canvas output contains imagery in an HDR color space and range.

[sunag]

Certainly those two concepts are distinct.

Yes, and this distinction is also made at the WebGPU level.

three.js/src/renderers/webgpu/WebGPUBackend.js

Lines 267 to 269 in f41b94a

	toneMapping: {
	mode: toneMappingMode
	}

https://developer.mozilla.org/en-US/docs/Web/API/GPUCanvasContext/configure#mode

In my mind, the term target screen output range seems clearer than the imaginary HDR range because of the canvas configuration definition, although I perfectly understand what you meant and agree, because we don't have a capabilities API and that wouldn't be the intention.

If we were to strictly follow the definition of WebGPU as seen in the code above for WebGPURenderer, it would be something like:

renderer.toneMappingMode = THREE.EXTENDED;

I think this doesn't quite fit with what we use, since we could use three.js in HDR screen output without defining any .toneMapping by default.

Defining .outputType to set the only screen target range seems more generic, and therefore more complementary use with .outputColorSpace and .toneMapping and in the canvas configuration and allow ease warning the user about incorrect setup at runtime.

I think workingBufferType is still the best name for the old colorBufferType, because all related buffers(internal) are dealing with workingColorSpace, so the color space conversion and tone mapping will only happen in the final stage when the output target is outputBufferType, for that reason in the PR the canvas type is called outputBufferType. This is consistent with outputColorSpace.

---

So we will have 3 distinct properties for this proposals in #32508:

• workingBufferType which will be for internal frame buffer renderings. 16bpc by default
• outputType which defines the target of the screen output range. 8bpc by default.
• outputBufferType which allows to define the output buffer, or canvas, as you prefer. 8bpc by default.

If outputBufferType is undefined, it will use the settings of outputType if defined, making the use of HDR a single line of WebGPURenderer but still maintaining all the aspects mentioned.

const renderer = new THREE.WebGPURenderer( {
	outputType: THREE.HalfFloatType
} );

And now allowing configurations such as:

const renderer = new THREE.WebGPURenderer( {
	outputType: THREE.UnsignedByteType,
	outputBufferType: THREE.HalfFloatType
} );

or

const renderer = new THREE.WebGPURenderer( {
	outputType: THREE.HalfFloatType,
	outputBufferType: THREE.HalfFloatType,
	workingBufferType: THREE.UnsignedByteType
} );

[donmccurdy]

If we were to strictly follow the definition of WebGPU as seen in the code above for WebGPURenderer, it would be something like:

renderer.toneMappingMode = THREE.EXTENDED;

I think this doesn't quite fit with what we use, since we could use three.js in HDR screen output without defining any .toneMapping by default.

Agreed that it's probably not a great fit. The other issue is that the "extended" tone mapping basically means "clip to the display limit" which is not much use when we don't know that limit and can't tonemap or color grade to it. So there's work to do on the WebGPU side there, and TBD what that will look like, but possibly gpuweb/gpuweb#5294.

It seems we are a bit stuck on the outputType/outputBufferType question, though, I don't agree that an option defined in terms of "half float" can safely be interpreted to enable HDR output. One more related comment in #32508 (comment).