Promote Node-based Materials to Core and Polish

[bhouston]

Description of the problem

@sunag did a great jobs on the node-based materials. While he got some criticism for his PRs when he was submitted them, usually dealing with internal design of the code compiler, I've always thought his work was amazing.

We have adopted node-materials for our rendering recently. We have found that it leads to more artistic creativity. It also leads to less complex code base -- the current Uber materials in Three.Js are just insanely complex and hard to maintain.

Thus I would propose that we move the node-based materials into core and then we slowly move most of the examples to use the node-based materials instead of the complex do-everything uber materials.

It is a big change, but I think it moves Three.JS forward.

Some of the changes I think are needed in order to make node-based materials first class citizens of Three.JS:

• add ability to get specialized layers out of the node-based materials easily (normal, depth, velocity, etc.) There is both an easy way and a hard way to do this. We'll start with the easy way.
• add more advanced options to the BRDF/EDFs such as anisotropy, basic sub-surface scattering.
• add ability to read AO from a pre-pass in the BRDF/EDF so it can only affect indirect diffuse rather than being layered on top and add an example of this.
• rework the texture node a bit, it has some problematic dependencies that make the graph difficult to evaluate. It was suggested by @jojobyte of NVIDIA to follow the model of UE4's texture node.
• have a separate clearcoat normal: MeshPhysicalMaterial: clearcoat should not use underlying normal map #12867
• add trilinear mapping example using the node-materials: Adding triplanar mapping #15048
• add displacement map to the node-based BRDF/EDF (which would require a vertex shader change.)
• add bump to normal example using the node-materials.
• add transmission factor to the root BRDF/EDF and an example to three.js: Feature: Add transmissionFactor to Materials #15941
• makes it easy to support arbitrary UV channels per map in a very simple way. Lots of power and flexibility. Thus negating the need for this: Support per-map per-Material UV channels, UV offset/repeat and texel scale/offset #8278
• allows for supporting world to UV mapping and other forms of procedural UV transforms.

This ties into my current proposal to glTF to adopt node-based materials:

https://docs.google.com/document/d/1Y6JFE2FV164IFDe7_cYhp2gzhSapB76fUNPgmsI6DDY/edit

I am not the first to suggest this but I can not find the previous discussion. I think it is just a matter of time until this happens, so might as well do this now. No other JavaScript 3D renderer has fully node-based materials. ;)

/ping @WestLangley @donmccurdy

Three.js version

• Dev

Browser

• All of them

OS

• All of them

Hardware Requirements (graphics card, VR Device, ...)

[Mugen87]

At some point the project will start to work at WebGPURenderer. How about starting with a node-based approach in this context and use the time now for other topics like adding support for remaining WebGL 2 features, further enhance light probes, enhance amount of JSM modules, focus on XR related stuff and the project support in general.

I personally would leave the material system for WebGLRenderer as it is and use WebGPURenderer as an opportunity to make some clear cuts/major changes like introducing a new material system.

[sunag]

hi @bhouston, thank you very much for this incentive...

Did you ever see the MeshStandardNodeMaterial?
https://github.com/mrdoob/three.js/blob/dev/examples/js/nodes/materials/MeshStandardNodeMaterial.js
https://github.com/mrdoob/three.js/blob/dev/examples/js/nodes/materials/nodes/MeshStandardNode.js

My idea is finish this material and test with all examples of threejs until normalizing the code replacing the references, for example:

// import three.js lib
THREE.MeshStandardMaterial = THREE.MeshStandardNodeMaterial;
// start/test example

And do this with the others materials...

I would what you think of this node because it is based on your shaders:
( referring more specifically to the code design )
https://github.com/mrdoob/three.js/blob/dev/examples/js/nodes/misc/TextureCubeUVNode.js
This is the most complex node so far and the shader was a fork of the core.

add more advanced options to the BRDF/EDFs such as anisotropy, basic sub-surface scattering.

Did you think any way to add anisotropy in physical materials? Or a path you recommend.

add bump to normal example using the node-materials.

I think it was done here:

three.js/examples/webgl_materials_nodes.html

Lines 987 to 988 in 93e72ba

	//bumpMap.toNormalMap = true;
	//mtl.normal = new THREE.NormalMapNode( bumpMap );

rework the texture node a bit, it has some problematic dependencies that make the graph difficult to evaluate. It was suggested by @jojobyte of NVIDIA to follow the model of UE4's texture node.

What exactly is that? Currently there is a cache system to reuse texture samples in the same UV.

makes it easy to support arbitrary UV channels per map in a very simple way. Lots of power and flexibility. Thus negating the need for this: #8278

I think something has been done in this sense here:

three.js/examples/webgl_materials_nodes.html

Lines 912 to 917 in 93e72ba

	var translate = new THREE.Vector2();
	var rotate = 0;
	var scale = new THREE.Vector2( 1, 1 );
	var texture = new THREE.TextureNode( getTexture( "brick" ) );
	texture.uv = new THREE.UVTransformNode();

[bhouston]

@sunag I believe strongly your stuff belongs in core. :)

Did you think any way to add anisotropy in physical materials? Or a path you recommend.

Clara.io has supported anisotropic roughness since 2014. I want to contribute it back to three.js but I hate modifying our horribly complex uber shaders -- very few people understand the uber shaders because their complexity is so high, so many nested defines spread everywhere.

Moving to node-based will simplify Three.JS's core so we can move to the next level.

Did you ever see the MeshStandardNodeMaterial?

I forgot about this, but yes, this is the easy way to move this to core! :) And then we can deprecate the old one and use this instead. I bet it could make ThreeJS smaller in terms of its code base or at least keep it the same size. Thus Three.JS is better than ever other open source JS-based 3D rendering library and it remains the same code size.

What exactly is that? Currently there is a cache system to reuse texture samples in the same UV.

I did reference the wrong person, it wasn't jojobyte, but rathan Jan Jordan the NVIDIA MDL product manager. Basically he suggested a T2V type that is passed around. This allows for some different node structures that he says are better. I can explain later as I want to keep this discussion focused.

makes it easy to support arbitrary UV channels per map in a very simple way. Lots of power and flexibility. Thus negating the need for this: #8278

I think something has been done in this sense here:

With regards to arbitrary UV channels, I know that your system supports it. :)

[bhouston]

At some point the project will start to work at WebGPURenderer. How about starting with a node-based approach in this context and use the time now for other topics like adding support for remaining WebGL 2 features, further enhance light probes, enhance amount of JSM modules, focus on XR related stuff and the project support in general.

Why delay? @sunag made a system that works with WebGLRenderer right now. It is just a matter of adopting it. Sunag has made it relatively easy. I know there is still a bunch of work to polish it up and have it work correctly with the various post effect scenarios, etc. I can commit to doing this if we do it in the relatively near term. I've done a few heavy lift PRs in the past such as redesigning the lighting system and the animation system, so generally I can pull these off:

#7324
#6934

[Mugen87]

I'm just concerned about backwards compatibility and the necessary time effort to convert the related example code which is quite extensive to be honest. Just wanted to point out a different approach that might be easier to handle and is maybe more strategic.

[zadvorsky]

So does this mean that the built-in materials would be replaced by node materials entirely, and user custom node materials would be treated the same way in the renderer?

[WestLangley]

So does this mean that the built-in materials would be replaced by node materials entirely

No, the built-in materials will be implemented as node materials.

[bhouston]

I'm just concerned about backwards compatibility and the necessary time effort to convert the related example code which is quite extensive to be honest. Just wanted to point out a different approach that might be easier to handle and is maybe more strategic.

The idea is that we polyfill MeshStandardMaterial such that it uses the node-based system to compile while keeping the current interface. Thus we get rid of the insane set of defines and what not that our uber materials utilize and instead we use the node-based compiler system of sunag. Thus it is work, and I am sure we will find bugs, but it isn't impossible.

[donmccurdy]

Looking at the overall plan, getting the uber-materials out of the renderer codebase is big. Ideally the renderer should have no dependencies on either the uber-materials or the node-based materials. If I'm only using a small custom ShaderMaterial, the entire threejs material system (nodes or otherwise) would ideally be tree-shakeable. Is that a realistic goal?

---

Some of the changes I think are needed in order to make node-based materials first class citizens of Three.JS...

Could we split this list, to identify tasks that would definitely be required up front? Reworking the texture node and full AO support might be candidates.

add bump to normal example using the node-materials.

Does this work with a procedural bump map?

---

Just to mention them here, a couple other issues I've run into with the current implementation:

1. Defining a custom FunctionNode is tricky. A typical "noise" function may call other functions, or have overloaded versions of itself with different parameter types. The regex that parses FunctionNode for a function name breaks on this and returns ''. Is there a safe way to import something like a library of custom noise functions? Example.
2. Organization of the Math nodes by number of arguments (Math1Node, Math2Node, ...) has bit me multiple times. If I'm writing a node graph imperatively, I'd prefer to think about what the node does rather than the number of arguments it has. Would a consolidated MathNode be possible?
3. Optimizations, mentioned by @zadvorsky in a previous discussion:

The built-in three.js materials (standard, phong, etc) are implemented differently from Shader Materials. They have optimizations for WebGL state changes that do not apply to Shader Material. Without these optimizations, node based materials lose a lot of their oomph if you have a complex scene graph.

^This also applies to NodeMaterials.

[WestLangley]

node based materials lose a lot of their oomph if you have a complex scene graph

"oomph". Can you please explain what you are referring to?

[donmccurdy]

I believe the issue was that in a scene reusing many otherwise-identical materials with different uniform values, state change optimizations that would apply to default materials do not apply to ShaderMaterial.

[zadvorsky]

If I understand the current system correctly, having two identical Shader Material instances (in terms of vert and frag shader) with different uniforms would still cause a full useProgram switch, which is expensive, whereas having two MeshStandardMaterial instances would only update the uniforms (aside from differences that affect defines).

Any new system should, ideally, give custom materials the same optimizations as the build in materials.

[sunag]

Organization of the Math nodes by number of arguments (Math1Node, Math2Node, ...)....

possible and interesting...

Optimizations, mentioned by @zadvorsky in a previous discussion:

Considering that I understood the question... If use an differents material with same inputs and types but diferents nodes, threejs would share the same program without problems like this benchmark show:
https://threejs.org/examples/?q=nodes#webgl_performance_nodes

Defining a custom FunctionNode is tricky

That is a good question...

The correct approch today for multiples functions is this:

three.js/examples/js/nodes/misc/TextureCubeUVNode.js

Lines 137 to 158 in 51afa0b

	var textureCubeUV = new FunctionNode( [
	"TextureCubeUVData textureCubeUV( vec3 reflectedDirection, float roughness, in float cubeUV_textureSize ) {",
	" float roughnessVal = roughness * cubeUV_maxLods3;",
	" float r1 = floor(roughnessVal);",
	" float r2 = r1 + 1.0;",
	" float t = fract(roughnessVal);",
	" vec2 mipInfo = MipLevelInfo(reflectedDirection, r1, roughness, cubeUV_textureSize);",
	" float s = mipInfo.y;",
	" float level0 = mipInfo.x;",
	" float level1 = level0 + 1.0;",
	" level1 = level1 > 5.0 ? 5.0 : level1;",
	"",
	// round to nearest mipmap if we are not interpolating."
	" level0 += min( floor( s + 0.5 ), 5.0 );",
	"",
	// Tri linear interpolation."
	" vec2 uv_10 = getCubeUV(reflectedDirection, r1, level0, cubeUV_textureSize);",
	" vec2 uv_20 = getCubeUV(reflectedDirection, r2, level0, cubeUV_textureSize);",
	"",
	" return TextureCubeUVData(uv_10, uv_20, t);",
	"}"
	].join( "\n" ), [ TextureCubeUVData, cubeUV_maxLods3, MipLevelInfo, getCubeUV ] );

Create a FunctionNode for each GLSL function and add the depedencies using includes argument of FunctionNode.

three.js/examples/js/nodes/core/FunctionNode.js

Line 12 in 93e72ba

function FunctionNode( src, includes, extensions, keywords, type ) {

I am thinking of starting the dev of a threejs shader language parser using GLSL syntax:

var tjslNodes = new TJSLNode(`{

vec3 hash(vec3 p) { return vec3( 1.0 ); }

vec3 voronoi3D(const in vec3 x) { 
return hash( x );
}

}`);

// this would parse the code and return it in nodes

tjslNodes.methods.voronoi3D // return FunctionNode with hash function depedencie
tjslNodes.methods.hash // return FunctionNode no depedencies

There is a care in this process for a better automatic optimization and auto rename system(avoiding name conflict).

And for example most of the root nodes like TextureCubeUVNode could based on proper shader language that turned the whole hierarchy into nodes.
Not equal but similar to what happens in: https://github.com/mrdoob/three.js/tree/dev/src/renderers/shaders/ShaderChunk)

[bhouston]

If I understand the current system correctly, having two identical Shader Material instances (in terms of vert and frag shader) with different uniforms would still cause a full useProgram switch, which is expensive, whereas having two MeshStandardMaterial instances would only update the uniforms (aside from differences that affect defines).

I think that is easy to fix. Just have a content hash of the shader network that is independent of the uniforms and use that as a lookup into a shader cache of some sort. This would save both compilation as well as switching costs. This is a fixable problem and a straight forward optimization I think.