Questions about VideoTexture's new approach

[ycw]

Description

Related: #31416

When sampling VideoTexture using importExternalTexture, GPU automatically converted values to linear-sRGB, but after switching to copyExternalImageToTexture with texture format set to rgba8unorm, sampling returns non-linear sRGB-encoded texels without decoding... it's unclear whether this is an intended change or a misconfiguration?

Reproduction steps

n/a

Code

n/a

Live example

n/a

Screenshots

No response

Version

dev

Device

No response

Browser

No response

OS

No response

[Mugen87]

The format of the video texture depends on the colorSpace property. If set to THREE.SRGBColorSpace like below, the format ends up as rgba8unorm-srgb which means the texels are decoded to linear-srgb.

three.js/examples/webgpu_video_panorama.html

Lines 70 to 71 in 3b1ff76

	const texture = new THREE.VideoTexture( video );
	texture.colorSpace = THREE.SRGBColorSpace;

[ycw]

The format of the video texture depends on the colorSpace property. If set to THREE.SRGBColorSpace like below, the format ends up as rgba8unorm-srgb which means the texels are decoded to linear-srgb.

Got it, so we should make VideoTexture#colorSpace always be SRGBColorSpace for the new approach, so that Engine would create a rgba8unorm-srgb GPUTexture which in turns sampled texel is always decoded to linear-srgb, agree?

[Mugen87]

If the video holds sRGB values, yes. I'm not sure that's true for every video outside there though.

[ycw]

If the video holds sRGB values, yes

Not really. Using copyExternalImageToTexture with colorSpace:'srgb' causes the texture to store sRGB-encoded values. If the texture format is not *-srgb, the GPU won't decode it during sampling, and you'll get sRGB values instead of the expected linear values. This behavior doesn't depend on the video-decoded-pixels color space. The spec ensures that when both the source and destination are sRGB, double encoding won't happen.

In short, since the new VideoTexture approach uses the copyExternalImageToTexture API, if a *-srgb texture format isn't properly used, there's a high chance the colors will be incorrect during shader processing, unless the shader manually performs a conversion. However, based on my current testing, I don't think such a conversion is being done.

https://ycw.github.io/three.js/examples/demo1.html, Left:NoColorSpace, overly-shifted from the video's colors. Right:SRGBColorSpace, relatively close to the video's colors(it's not exact same because the light intensity is arbitrarily set to 5), the key point is that you've already sensed the sampling problems that arise from using *-srgb versus non *-srgb texture formats upon copyExternalImageToTexture.

[Mugen87]

You are right. We are currently ignoring the colorSpace property of the GPUCopyExternalImageDestInfo interface. Since the only valid values are "srgb" and "display-p3" and the default is "srgb", it seems all video textures must set their colorSpace property to THREE.SRGBColorSpace.

Would you be open a PR that fixes this issue in webgpu_materials_video and webgpu_lights_projector?