Allow users to provide a custom shadow shader

[Ansraer]

Describe the project you are working on

3D foliage rendering

Describe the problem or limitation you are having in your project

When rendering a scene godot does 5 passes of it: One pre depth pass, three shadow passes, and finally a color pass.
With Godot's default PBR shader the depth and shadow pass use a pretty bare bones fragment shader since only the depth output is wanted.

However, for custom shaders this is not the case. While Godot still creates a shadow pipeline for the custom shader it uses the exact same fragment code as the color pipeline. This means that the depth and shadow passes of custom shaders calculate a lot of unnecessary output, wasting valuable time.

Some tests with Renderdoc have shown that the depth pass for a very basic noise based material with alpha scissors can take up to 60 microseconds. By manually trimming the shader down to only what is necessary for depth rendering (the alpha scissors) I managed to cut down the execution time to 20 microseconds.

With more complex custom shaders the performance loss would be even greater.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

What I want is the ability to provide a custom shadow shader. As far as I can tell Godot already has code for using a different shader for shadow and depth pass:

https://github.com/godotengine/godot/blob/e80aedbf20a7846a0e21dc8623823eef10b2e676/servers/rendering/renderer_rd/forward_clustered/render_forward_clustered.cpp#L2669-L2683

All that would be needed is a way to expose this to the user.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Similar to the next pass property users would be able to set another shader as a shadow material. With some modifications to the code linked above it should be possible to use that material for shadow rendering.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Not that I am aware of. I suppose you could turn off shadows and create a duplicate that only renders shadows with a different material? But that would come with its own problems.

Is there a reason why this should be core and not an add-on in the asset library?

rendering is core.

[wojtekpil]

We have currently a section for shadow meshes in inspector. Those meshes have a material slot which could be used for this purpose. Currently it is a little bit counter intuitive, because overriding this does nothing.

[Ansraer]

@wojtekpil Maybe I am blind, but where are shadow meshes exposed to the editor in godot 4.0? Afaik Godot creates shadow meshes automatically on import. Also, based on my understanding of the code, godot uses a shared material for those shadow meshes.

My idea was to add a field below next pass like in the mockup below:

AwesomeMockup

[wojtekpil]

Under the mesh menu

I did not read the current code so I can't tell how does it work under the hood. It's just the place where I expected to replace a material for a shadow mesh.

[Ansraer]

The problem with this shadow mesh is that it is only used as long as you don't use a custom shader and edit anything vertex, position or alpha related. If you look at the code I linked in my initial post you can see how it isn't used once you edit one of those.
Based on what I can see this is mostly intended to be used by importers that auto-generate simplified meshes.
I wrote a quick PoC PR, here is how the UI currently looks:

PoC

This uses a shadow shader where the left part of the mesh isn't rendered and this doesn't cast any shadows.

[clayjohn]

We discussed this at the proposals meeting today, the consensus was that we should instead introduce an AT_SHADOW_PASS constant in the spatial shader that is true during the shadow pass and false otherwise. This would allow users to specify a different code path for shadows. We can rely on SSA to compile out unused code since it will be constant for the entire shader (either using a macro define or a specialization constant).

From a user perspective it would allow users to write things like:

void fragment() {
    if (!AT_SHADOW_PASS) {
         // do expensive transparency calculations here
    } else {
         // do cheap approximation with discard here
    }
}

[Ansraer]

We discussed this at the proposals meeting today, the consensus was that we should instead introduce an AT_SHADOW_PASS constant in the spatial shader that is true during the shadow pass and false otherwise. This would allow users to specify a different code path for shadows. We can rely on SSA to compile out unused code since it will be constant for the entire shader (either using a macro define or a specialization constant).

From a user perspective it would allow users to write things like:

void fragment() {
    if (!AT_SHADOW_PASS) {
         // do expensive transparency calculations here
    } else {
         // do cheap approximation with discard here
    }
}

Ah damn it, I keep missing the meetings.
Yeah, this will probably be a far better solution than what I currently use in my fork.
Since the shadow shader is also used for the pre depth pass it might be a better idea to use something like DEPTH_ONLY (to better convey that this will be true for the depth and shadow pass), and have a second, non-constant, boolean to differentiate between shadow and pre depth pass.

[clayjohn]

Ah damn it, I keep missing the meetings. Yeah, this will probably be a far better solution than what I currently use in my fork. Since the shadow shader is also used for the pre depth pass it might be a better idea to use something like DEPTH_ONLY (to better convey that this will be true for the depth and shadow pass), and have a second, non-constant, boolean to differentiate between shadow and pre depth pass.

I wouldn't worry about exposing DEPTH_ONLY unless someone points out a use-case for a custom depth-prepass/shadow shader. It should be fine to just expose AT_SHADOW_PASS. It is important to do this with a #define or with a specialization constant so that the SSA optimizer can remove the code inside the branch. If you use a uniform or push constant there is no guarantee at compile time that the code won't run.

In the GLES3 backend, this can be done with a shader varient (which is roughly equivalent to a specialization constant)

[Ansraer]

Oh, I think you misunderstood what I meant @clayjohn.
I want DEPTH_ONLY to be the name of the define (or constant) that is only true for the depth pre-pass and shadowmap pipeline (internally this is MaterialData::shadow_pass afaik).
imo telling the user what the result of the shader is (in this case writing only to depth) makes more sense than calling it AT_SHADOW_PASS, when it is used for passes other than shadows as well.
And I get why it needs to be either a define or a specialization constant, after all the entire point of my proposal is to make sure that as much code as possible is removed before the shader is executed. Just wanted to propose a different name.

The reason why I proposed the second (not const) bool was so that people could manipulate only the shadows without affecting the depth pass. While not useful for my particular usecase I could totally see how someone working on a horror game might want to modify only the shadows without changing the depth or color pass.

EDIT: Now I really want to make a horror game with shadows that try to grasp the player somehow. Just imagine seeing a completely unassuming tree, and then you look down and notice how the shadows of its branches slowly creep towards you...
I should write that down for the next game jam I participate in.

[EMBYRDEV]

The reason why I proposed the second (not const) bool was so that people could manipulate only the shadows without affecting the depth pass. While not useful for my particular usecase I could totally see how someone working on a horror game might want to modify only the shadows without changing the depth or color pass.

+1 that would be a super useful feature and just increases engine flexibility.

[clayjohn]

The reason why I proposed the second (not const) bool was so that people could manipulate only the shadows without affecting the depth pass. While not useful for my particular usecase I could totally see how someone working on a horror game might want to modify only the shadows without changing the depth or color pass.

+1 that would be a super useful feature and just increases engine flexibility.

That's a plus one for being able to manipulate the shadow pass apart from the depth pre-pass? Or a plus one to the proposal in general? If the former, can you please elaborate?