RGBM texture banding on iOS

[vjevremovic]

Describe the bug
RGBM PNG textures on iOS are showing extreme banding.

To Reproduce
Run these examples on any iPhone/iPad
https://threejs.org/examples/#webgl_loader_texture_rgbm

https://threejs.org/examples/#webgl_materials_envmaps_hdr
(select RGBM map)

Platform
iOS iPhone/iPad

[Mugen87]

Please fill out the bug report template properly. Especially you should share a screenshot that shows the rendering issues.

[vjevremovic]

Will do that. By studying the forums it looks that it is related to the way iOS loads PNGs. Other formats like BASIS work ok, but then again they suffer from artifacts, especially in the alpha channel. I have no idea what would be the right solution.

[Mugen87]

I've tested this issue today on a iPad (8th generation) with iPadOS 14.3 and Safari. I can see the same artifacts like in the screenshot.

The RGBM example does look as expected on a Pixel 4a though. Besides, the RGBE/HDR example looks good on the iPad, too.

[mrdoob]

Do we still need RGBM? 🤔

[mrdoob]

/cc @elalish

[Mugen87]

AFAIK, encoding/decoding with RGBM is faster than RGBE. Besides, you can always use linear filtering which does only work with RGBE when using floating point textures.

[vjevremovic]

RGBM is essential for working with 8bit encoded HDR images. Alpha channel usage for holding noncolor data is also mega important.

[mrdoob]

But, can't those 8bit HDR images be converted into a RGBE DataTexture at load time?
That way we'll have less shader code paths.

[vjevremovic]

No.
If it is premultiplied after load you lose precision and get that banding from the screenshot.

[elalish]

It's true that RGBE is a bit more expensive than RGBM (since you have to do custom interpolation instead of the normal hardware version), but it also works much better, especially for very high dynamic ranges. RGBE does great with 8bit HDR images, but converting between RGBM and RGBE could easily have bugs. However, using hardware interpolation on RGBM is also wrong (just not quite as wrong as RGBE) and could easily be responsible for the banding you see. I would recommend removing RGBM support entirely, as the whole format feels like a hack to me.

[vjevremovic]

RGBM format is not a hack. It is a legitimate algorithm to pack high dynamic range data in 8bit per channel image. Banding occurs due to iOS natively premulitplying alpha on PNG loading. Premultiplying alpha channel is wrong in so many different ways, especially in 3D engines. I would suggest implementing a custom loader for PNG on iOS (based on libpng), as this would solve many other issues too.

[elalish]

Okay, but have you looked at how much error is introduced by doing naive 4-channel linear interpolation on correct RGBM data? It can get significant if the gradients are steep. Also, for any given float triple there are multiple ways to encode the value to RGBM, which makes consistency tricky.

[vjevremovic]

Of course limitations are expected when you are packing float point data in fixed point. No matter how you pack/unpack them what is important is that texture data that has alpha channel is tempered natively by iOS on loading PNGs, and that is not ok. Let's focus on that problem. RGBM or any other format (or algorithm) that is utilizing alpha channel in textures is in the same situation.

[Mugen87]

Is there a JavaScript port of libpng? As a start, we could use it to implement a RGBMLoader that resolves this issue.