NodeMaterial: Using native code ShaderLib approach

[sunag]

I have this idea recently, I will mature this, basically, this use 100% native shader code with support to NodeMaterial in some inputs that non use texture uniforms like: color, opacity, etc.

Its replace in runtime uniform vec3 diffuse to vec3 diffuse and add the code relative of color input after main(){ and relative node uniform, vars and etc. This approach is compatible with almost all features of NodeMaterial.

The implementation in core will be more simple since it does not modify the core shader but be more limited with relation at some inputs.

Is necessary change the uniform logic inside of WebGLRenderer.

Final code should be:

// the end result should be
var material = new THREE.MeshStandardNodeMaterial();

// replace "uniform vec3 diffuse" to "vec3 diffuse"
// add functions code in top of shader and relative code after main()
var uvTransform = new Nodes.UVTransformNode();
uvTransform.setUvTransform( 0, 0, 50, 50, 0 );

material.color = new Nodes.CheckerNode( uvTransform );

Live here:
https://raw.githack.com/sunag/three.js/dev-node-native-live/examples/index.html?q=variations#webgl_materials_variations_toon

WIP

/cc @bhouston @mrdoob @WestLangley @Mugen87 @donmccurdy ...

[sunag]

[sunag]

New node uniforms:

[sunag]

This approach is very simple to implement extending natives materials with Nodes using NodeBuilder instead of convert natives material in NodeMaterial based, this last will be a big change.

Using this approach we can add includes relative at NodeMaterial for the Nodes inputs like normal:

meshphysical_frag.glsl.js

...
	#include <normal_fragment_begin>
	#include <normal_fragment_maps>
	// used for index the snippets of the NodeMaterial normal code
	#include <normal_fragment_nodes>
...

Final user:

// native material
var material = THREE.MeshStandardMaterial();

// node extension
material.normal = new Nodes.NormalMapNode( new Nodes.TextureNode( texture ) );

THREE.MeshStandardMaterial using NodeBuilder port ( this is a draft code only ):
https://github.com/mrdoob/three.js/blob/ae11f5fa429110c198a5887bf95001c48ea383e4/examples/jsm/nodes/materials/MeshStandardMaterial.js

[donmccurdy]

Would it be much trouble to include the suggested changes to the core library (just WebGLRenderer?) as well?

I am having trouble understanding the tradeoffs here — do you agree with these claims about the PR?

1. Users of nodes/* would attach nodes to native threejs materials, but nodes/* would not create new Material subclasses of its own.
2. This could be supported on all threejs material types (MeshBasicMaterial, MeshStandardMaterial, ...).
3. The code would be easier to introduce and to maintain with this approach.

On (3), in particular, I cannot guess by looking at the PR. 😇 Are there other pros/cons I've missed?

This approach is compatible with almost all features of NodeMaterial.

Which features do you think would be incompatible? Or which inputs become more limited?

It sounds like the proposal would also mean adding various properties to the Material classes, so for example users might choose between:

// Used only if .roughnessNode is not set?
material.roughness = ...;
material.roughnessMap = ...;

// If set, both .roughness and .roughnessMap are ignored?
material.roughnessNode = ...;

I would be curious what @mrdoob thinks of this API? Or would the user set a node on the existing .roughness or .roughnessMap properties? Not needing new material classes does sound nice...

Finally, do you think it would be possible to implement this without having the core material classes actually import anything from nodes/? For example:

// my-application.js

import { FloatNode, NodeBuilder } from 'three/examples/jsm/nodes/nodes.js';

material.roughnessNode = new FloatNode( 0.5 );
material.setNodeBuilder( new NodeBuilder() );

// MeshStandardMaterial.js

MeshStandardMaterial.prototype.build = function ( shader, renderer ) {
	
	const main = 'void main() {';
	
	const builder = this.nodeBuilder;
	const nodes = builder.createNativeNode( 'standard' );

	nodes.color = this.color;

	builder.setMaterial( this, renderer );
	builder.build( nodes, nodes );

	...

}

I don't know exactly what we decided about all of nodes/ moving into the core library, but the possibility of having it be optional is worth considering.

[sunag]

1 - Users of nodes/* would attach nodes to native threejs materials, but nodes/* would not create new Material subclasses of its own.
2 - This could be supported on all threejs material types (MeshBasicMaterial, MeshStandardMaterial, ...).
3 - The code would be easier to introduce and to maintain with this approach.

[1,2,3] Yes, with all of them. And about (2) include ShaderMaterial.

Or would the user set a node on the existing .roughness or .roughnessMap properties?

Its replace in runtime uniform vec3 roughness to vec3 roughness. With maps inputs like .roughnessMap being a sample2d is not compatible with this approach, but its indifferent once that we can use TextureNode inside .roughness property for this case.

Which features do you think would be incompatible? Or which inputs become more limited?

The StandardNodeMaterial has an input list that is more simplified, for example: .environment not need .envMapIntensity, and .lightMap not need .lightMapIntensity and so on.

Before of this commentary ( #18162 (comment) ), adding includes relative at Nodes in native shader some limitations that I imagined have been resolved. For example, add a #include <lights_fragment_nodes> and a new property like .environment or .env in MeshStandardMaterial for a node based environment light.

I don't know exactly what we decided about all of nodes/ moving into the core library, but the possibility of having it be optional is worth considering.

I think that we can add a minimalist code of NodeMaterial in core like NodeBuilder and Node to serialization and optimize to the core, and others more specifics stay in examples.

It sounds like the proposal would also mean adding various properties to the Material classes, so for example users might choose between:

Its is interessating, I think initialy detect and replace native uniforms in case of use some node otherside of default value ( related #18175 )

// Add .roughness uniform if this is not Node ( default )
material.roughness = 1;

// .roughness uniform is replaced for a variable ( uniform property is removed )
material.roughness = new Nodes.TextureNode( texture );

[donmccurdy]

Thanks, the limitations you mention seem totally fine.

So the material properties would be something like:

class MeshStandardMaterial {
  color: Node | Color;
  map: Texture;
  opacity: Node | number;
  roughness: Node | number;
  roughnessMap: Texture;
  metalness: Node | number;
  metalnessMap: Texture;
  normal: Node;
  normalMap: Texture;
  ao: Node;
  aoMap: Texture;
  aoMapIntensity: number;
  environment: Node;
  envMap: Texture;
  envMapIntensity: number;
}

Is everyone OK with that API? @mrdoob @bhouston

It seems mostly clear and readable to me. But some existing code will obviously break if given a material that has nodes where primitives used to be, and support for nodes in places like OBJExporter and GLTFExporter will probably take a while.

I think that we can add a minimalist code of NodeMaterial in core like NodeBuilder and Node to serialization and optimize to the core, and others more specifics stay in examples.

Sounds good to me. If the nodes themselves don't have to be part of the core library that's definitely a benefit, so users can write their own Node classes.

[donmccurdy]

We'd need material.mask too, I guess... that one might be a bit harder?

[sunag]

We'd need material.mask too, I guess... that one might be a bit harder?

I think that .mask is not hard to implement, but maybe in this case we can use opacity: Node | number instead? ( This in together with ALPHA_TEST )

[donmccurdy]

Agreed, I forgot about .opacity but if we follow the structure above it should also accept a Node. I've added that to my earlier comment.

In addition to that, we might still want .mask: Node or .alphaTest: Node | number? Similar to #15654. But I guess the ALPHA_TEST uniform combined with opacity: Node might be enough for now.

[sunag]

Nodes is compatible with any material with this update, is only necessary add the properties of others materials in propertyNames of WebGLPrograms.

https://github.com/sunag/three.js/blob/93dfb3cddf8ce8c788f8aefa77fbd1e16de7b3a9/src/renderers/webgl/WebGLPrograms.js#L42-L44

After this PR ( #18399 ) some modification has necessary in WebGLPrograms and WebGLProgram.

This still is a WIP but I think that the way is this.

[sunag]

All material / variations node material test: Live here:

https://raw.githack.com/sunag/three.js/dev-node-native-live/examples/index.html?q=variations#webgl_materials_variations_toon

[sunag]

I think this issue is being better resolved and continued in the new node material system.
#20421