Shading models using incorrect view direction for orthographic projections.

[sciecode]

#17379 exposed a problem with the current orthographic behavior of light. However it was quickly dismissed as a help question, but I don't think this is the case.

It was thought to be a problem with physical and standard models, but I believe the real issue at hand is the incorrect calculation of the view direction for orthographic cameras.

Currently the view direction is being calculated as follows vViewPosition = - mvPosition.xyz;
However this is not a correct model for an orthographic view. This is supposed to be fixed as the camera direction, disregarding the object position.

This is causing multiple unexpected behaviours. Given a fixed orthographic camera and a scene being lit by a directional light, the specular light component shouldn't move if the object moves. This is not what's happening on Dev - dev example

From the image, you can see the fresnel component is being incorrectly placed on the object. As previously noted, the specular component is also incorrectly moving when it shouldn't be.

---

In order to fix this behavior, I just replaced the vViewPosition calculation for every shading model with the following:
vViewPosition = isPerspectiveMatrix( projectionMatrix ) ? - mvPosition.xyz : vec3( 0, 0, 1 );

This is the example of what should be actually happening - fix example

Would appreciate if other contributors, that are more familiar with orthographic projection, could confirm this is indeed the correct way of handling the issue at hand.

[WestLangley]

While you are thinking about this, see if you can also study reflections from an environment map.

It is not clear to me how we should be handling lighting with an orthographic camera.

It might be of interest to compare with other engines.

[alelepd]

Thanks for this @sciecode! Glad to know there was a fix.

I compared this fix with blender and they both look the same now.

Recorded a video from blender with similar settings:
https://jumpshare.com/v/kS3GlDUihE64q8AsmMdq

[sciecode]

I've studied a bit more about how orthographic projections are supposed to work.

It is a form of parallel projection, in which all the projection lines are orthogonal to the projection plane

The only real difference is that you can assume that lighting rays read by the camera are the ones with exact opposite direction from the camera view direction. You can think of it as if there was a filter, on top of the screen space, blocking any incident rays coming from lateral directions.

With that being said, the lighting calculations should stay exactly the same. Only the view direction calculations need to be adjusted accordingly.

---

I've implemented this behavior on environment reflections / refractions and the result looks correct now.

dev example
fix example

( Note that you can rotate the camera and both the environment reflections and the specular component still behaves as you would expect )

---

I've tried looking for how other engines are handling this behavior:

Babylon.js is incorrectly calculating the view directions exactly as we are on dev. example
Blender and Cinema4D, however, are behaving exactly as the proposed fix.

I have a partial PR ready, in case you guys want to proceed with this change. There's just one minor varying collision with vIsPerspective that needs to be solved, but I'll leave this discussion for this possible PR.

[sciecode]

There are a few non-intuitive side effects of these changes, however. Those should be carefully considered in favor of correctness versus clarity.

The concept of a CubeTexture skybox isn't really applicable to an orthographic projection. The reason for that is because the objects (even from afar) should look real size. Which means the orthographic camera would only capture an infinitesimal fraction of the "background" and expand it to fill all the screen. This is also why the current skybox doesn't and shouldn't work in orthographic mode.

This is manageable, for the most part, as long as users are aware of this limitation.

This does not mean that the reflection model is incorrect when using an environment map, because although the camera only captures orthogonal rays coming from the object, the accumulation of light and reflection happens in a perspective world. So the result is still acceptable and correct.

Which is not the case for the refractive model. The accumulation of light is coming from what would've been this stretched infinitesimal skybox, so not only the refraction looks absurdly enlarged, but it also doesn't look very good at all.

This is still the correct behavior, but it makes me wonder if it is something we are actually ok with.

[alelepd]

That's interesting. Have you tried displaying the cube map in the background to see how the refraction behaves?

I made some tests in c4d that might help to compare:
(this is a sphere with 1.01 IoR and a big cube geometry with a cube map, using a near clipping to be able to see within the cube using an orthographic camera).

[sciecode]

This is the point I'm getting at @alelepd.

There's no correct skybox implementation for an orthographic view, this is a perspective skybox with arbitrary parameters.

Although the specular highlights and reflections work well using the correct orthogonal view direction, Cinema4D, at least, fallback to this incorrect perspective view direction for refractions. This is a compromise between correctness and visual appeal.

Were the case they were using the correct calculations, not only would the refraction be arbitrarily enlarged, but it would also not change based on the object position on the screen. Much like the specular and reflection components.

[alelepd]

Actually, I am not sure if there is anything special about an Orthographic camera and a cube skybox, after all, a skybox can be just a big geometry (cube, sphere, etc) with a texture mapped in a way that matches the geometry and this generates a visual effect of a 360 degrees view.

For example, in the first video from my last post, I was just moving the camera around a sphere inside the cube with a cube map, the refraction is not getting any perspective distortion. I purposely rotate the cam around the Y-axis and thus it looks like a perspective camera (but is not).

Here you can see what's going on:

If I move the ortho camera more freely (and zoom out), you can see the deformation around the edges of the cube and the "skybox effect" is no that good looking (this is less noticeable on perspective camera due to the camera intrinsic visual deformation).

Another way of seeing the difference between a perspective camera and an orthographic camera and how the refraction would look is by animating the perspective to an orthographic:

[sciecode]

This is outside the scope of what is discussed, the fact that you can use a cube box as a skybox for an orthographic projection does not make it visually correct, as demonstrated by your second gif.

Which is why I referred to it as a perspective skybox with arbitrary parameters, cause that's what it is. Users may implement whichever skybox model they see fit, we need to address the fundamental lighting model used by orthographic projection. Let's try not to derail from that.

[EliasHasle]

In https://discourse.threejs.org/t/coding-jam-pixel-perfect-spheres-without-high-res-geometry/10154, for the orthographic branch I assume the view "ray" to be defined per pixel as pointing from the pixel along the normal of the orthographic projection plane. I don't see how that would not in principle be compatible with environment map reflections/refractions.

In order to fix this behavior, I just replaced the vViewPosition calculation for every shading model with the following:
vViewPosition = isPerspectiveMatrix( projectionMatrix ) ? - mvPosition.xyz : vec3( 0, 0, 1 );

Shouldn't that else clause be vec3( 0, 0, -mvPosition.z ), to allow dependence on distance for ortho too, e.g. in fog?

Anyway, I am really not sure it is right to redefine view position like this. I think it is better to handle the peculiarities of orthographic projection in the usage of view position instead, such as when calculating distance-based fog, which in the proposed implementations (including my own) use the length of the view position, even for ortho.

If we want to avoid branching in the fragment shader, we may perhaps in some cases be better off by branching cameras at compile time (#ifdef ORTHOGRAPHIC etc.)? This would also open up a bit for more exotic camera projections...

I think for distance fog I will be able to avoid branching by exploiting that projectionMatrix[2][3] is -1 for perspective and 0 for orthographic.