Draft : Implementing structures.

[ld-kerley]

Struct support in MaterialX

Introduction

The MaterialX (1.38) specification document allows for “Custom Data Types” (see page 8). Custom data types can specify <member> elements, effectively defining a custom structure. Once defined these custom structs can be used as types for input and output elements in a node or node definition in MaterialX documents.
Parsing the document is currently supported, but nothing else is. None of the MaterialX shader generation code provides any support for these custom types.

Proposal

Inspired by how OpenShadingLanguage deals with structs, I believe there is a reasonably easy path to adding support to MaterialX shader generation, in a language agnostic way. (Previously the project struggled to find a way to peformantly support this idea across all destination shader languages).

Effectively the idea boils down to implictly namespaced shader parameters being generated dynamically at shader generation time. Instead of a node generating a function where the parameters are one-to-one with the input ports on the corresponding node definition, when we encounter a custom struct type we expand that parameter out to a set of parameters prefixed with the struct name.

For example, the following custom type and node definition

<typedef name="myTexCoord">
  <member name="ss" type="float"/>
  <member name="tt" type="float"/>
</typedef>
  
<nodedef name=“ND_extractMyTexCoordS_float” type=float“>
  <input name="in" type="myTexCoord" value="{0.5,0.5}"/>
  <output name="out" type="float" defaultvalue="0.5"/>
</nodedef>

would generate something similar to the following psuedo-code, note the namespaced input parameters to the function. Note, the separator string here (__I__) is certainly a subject for discussion.

float extractMyTexCoordS(
  float in__I__ss = 0.5,
  float in__I__tt = 0.5
) 
{
   // implementation
} 

The shader generation code can be updated in a shader language agnostic way. Each time a struct is discovered when traversing the nodegraph each operation (connecting nodes, setting local values, etc) is just applied for each child <member> of the custom struct.

Rationale

This might not be the most obvious approach, but the reason we choose to take it anyway, is primarily that it allows the nodegraph to give the illusion of using structs, without imposing any potential performance penalty on the destination shader language by actually using structs. It also allows a potential future shader language to be adopted if it doesn’t support the concept at all. There is strong precedent for this approach in OSL, which is a production proven shading language used on 100’s of feature films.

Considerations

A few subtleties that are open for discussion.

• Do we allow support structs that contain other structs?
  • I think technically we could support this (OSL does), the namespace for the parameter names could just contain multiple scopes.
• Do we allow partially initialized structs when setting values on input ports?
  • In OSL it is legal to provide an initializer list where the number of members in the list is not complete to the number of member in the struct, in that case the later members just remain unset (or default).
• Do we want to extend the specification to mandate that <member> elements contain a defaultvalue element to ensure nothing is ever uninitialized?

Proposed Specification

I’ve written what I would like to propose to the ASWF project as the update to the MaterialX specification for 1.39.

---

Custom Data Types

In addition to the standard data types, MaterialX supports the specification of custom data types for the inputs and outputs of shaders and custom nodes. This allows documents to describe data streams of any complex type an application may require; examples might include spectral color samples or compound geometric data. The structure of a custom type's contents is described using a number of <member> elements.

Types can be declared to have a specific semantic, which can be used to determine how values of that type should be interpreted, and how nodes outputting that type can be connected. Currently, MaterialX defines three semantics:

• "color": the type is interpreted to represent or contain a color, and thus should be color-managed as described in the Color Spaces and Color Management Systems section.
• "shader": the type is interpreted as a shader output type; nodes or nodegraphs which output a type with a "shader" semantic can be used to define a shader-type node, which can be connected to inputs of "material"-type nodes.
• "material": the type is interpreted as a material output type; nodes or nodegraphs which output a type with a "material" semantic can be referenced by a in a .

Types not defined with a specific semantic are assumed to have semantic="default".

Custom types are defined using the <typedef> element:

  <typedef name="manifold">
    <member name="P" type="vector3"/>
    <member name="N" type="vector3"/>
    <member name="du" type="vector3"/>
    <member name="dv" type="vector3"/>
  </typedef>

Attributes for <typedef> elements:

• name (string, required): the name of this type. Cannot be the same as a built-in MaterialX type.
• semantic (string, optional): the semantic for this type (see above); the default semantic is "default".
• context (string, optional): a semantic-specific context in which this type should be applied. For "shader" semantic types, context defines the rendering context in which the shader output is interpreted; please see the Shader Nodes section for details.
• inherit (string, optional): the name of another type that this type inherits from, which can be either a built-in type or a custom type. Applications that do not have a definition for this type can use the inherited type as a "fallback" type.
• hint (string, optional): A hint to help those creating code generators understand how the type might be defined. The following hints for typedefs are currently defined:
  • "halfprecision": the values within this type are half-precision
  • "doubleprecision: the values within this type are double-precision

Attributes for <member> elements:

• name (string, required): the name of the member variable.
• type (string, required): the type of the member variable; can be any built-in MaterialX type; using custom types for <member> types is not supported.
• defaultvalue (string, optional): the value the member variable will be initialized to if not explicitly set on a node.

If a number of <member> elements are provided, then a MaterialX file can specify a value for that type any place it is used, using an aggregated initializer, a semicolon-separated list of numbers and strings surrounded by braced. The expectation is that the numbers and strings between semicolons exactly line up with the expected <member> types in order. For example, if the following <typedef> was declared:

  <typedef name="exampletype">
    <member name="id" type="integer"/>
    <member name="compclr" type="color3"/> 
    <member name="objects" type="stringarray"/> 
    <member name="minvec" type="vector2"/> 
    <member name="maxvec" type="vector2"/>
  </typedef>

Then a permissible input declaration in a custom node using that type could be:

<input name="in2" type="exampletype" value="{3; 0.18,0.2,0.11; foo,bar; 0.0,1.0; 3.4,5.1}"/>
  

Once a custom type is defined by a <typedef>, it can then be used in any MaterialX element that allows "any MaterialX type"; the list of MaterialX types is effectively expanded to include the new custom type.

The standard MaterialX distribution includes definitions for four "shader"-semantic data types: surfaceshader, displacementshader, volumeshader, and lightshader. These types are discussed in more detail in the Shader Nodes.

[ld-kerley]

The code here is far from complete - but is posted to initiate a conversation around if this direction feels like the right one to go in. The example in the experimental folder should be renderable, and demonstrates the generated shader code.

[niklasharrysson]

Thanks for this great proposal @ld-kerley

The proposed solution is elegant in the sense that it's language agnostic. By restructuring the ShaderNodes at ShaderGraph construction time it requires minimal changes to existing shader generators.

But consequently it forces this solution on all existing and future languages, so a question is if that is ideal? For languages like OSL and MDL, that supports struct types, would we not want to have the MaterialX types mirrored in the generated code? The pros I can see is that maybe these languages, or future languages, can handle structs better internally than splitting them out to separate inputs. Also, from a shader interface point of view it feels more intuitive to get a shader out where the MaterialX types are represented as is, when structs are supported in the language.

One alternative solution would be to have the struct represented as a nesting of ShaderInputs, where the struct ShaderInput remains but with child inputs added for each member. Then each generator can choose how to handle it: either generate a single struct input or split into member inputs.

General question: I assume there are no plans to support connections to/from individual struct members? Instead we could have combine and extract nodes to support that, to keep it simple.

[kwokcb]

I am curious for thoughts on how integrations which do not use code generation parse these and handle interop?

For instance, I'm working on the Khronos glTF representation of graphs and there is no way to represent a struct there, or even any notion of type definitions. It's "simple" types likes floats, ints and tuples / matrices of these. Things like mixing in string types, inheritance are also issues since they require run-time evaluation.

Another connection point is naturally OpenUSD and how struct interop would work there?

Throwing out the idea to move the translation ("flatten") of structs into non-structs outside of code generation.
Then essentially code generation and other integrations do not need to be modified to support structs.
I can see this working for non-definitions but what would happen for a definition ? (nodedef) as translation can mean changing the input interface.

[ld-kerley]

@niklasharrysson - I would propose we explicitly disallow connections to/from individual struct members. Having been a shader author myself in a past life, there have certainly been times when I have wanted to ensure my struct stays "whole", ie. isn't messed around with my someone else. And given we already have the extract and combine paradigm I think using that for structs feels like a very natural extension. In my prior shader network system we would selectively create those nodes, leaving some structs "safe" from external intrusion.

I can speak for OSL that there would be no performance penalty for this approach, as OSL would just perform a very similar operation on the shaders in the shader group when it's constructed. "A struct in OSL is just syntactic sugar" - Larry Gritz).

I'm not familiar with MDL, but I would observe that this proposal for the shader generation is doing nothing more than a shader author 'could' very reasonably do anyway.

I do like the idea of in the shader generator space there being a higher level object that the specific language shader generators could specialize on, so that if MDL really did want to retain the struct as a first class connection point it could. The idea with the code here was really just to demonstrate the idea. I guess we could provide something similar to what I have here (albeit cleaned up), as a default implementation in MaterialXGenShader, and then specialize from there.

@kwokcb - I think if an application/platform has decided to forgo using the shader generation, then they have decided to take on the burden of implementing the backend themselves. Then so long as we adequately specify what the expectations are, those platforms would be free to select whatever implementation suited them, but I don't think MateiralX would have any sort of other responsibility there. We could certainly outline in the specification this namespace parameter approach as a guidance.

As for OpenUSD, as far as I'm aware at the moment there is no sort of formal specification of the formatted data that UsdMatx creates when it transposes MaterialX in to USD, so we would be free to propose whatever addition we saw fit. As I understand things UsdMatx transposes the MaterialX data in to USD, and then HdMatx transposes it back to MaterialX to then call code generation. Then I would think it's sufficient to just provide a token or string attribute on the shader that would represent the struct port, and whatever connection or locally set value it has. That does have the similar assumption that we're using the MaterialX code generation again, but I think any renderer that isn't using that we can just provide a similar level of guidance as to what MaterialX expects an implementation to do.

I think we 'could' move the flattening process somewhere earlier, potentially as an optional stage, but it feels a little like pushing the peas around the plate. I would assume that you'd end up in a similar situation of having to effectively author new nodedef entries with the flattened inputs, and then those integrations that don't use code generation wouldn't have implementations for those definitions.

[niklasharrysson]

Fully agree on disallowing connections to individual struct members, and the extract/combine paradigm can be used selectively, to keep it clean and simple.

Yes I think it would be valuable to preserve the higher order structure in the ShaderGraph space, to let specific languages specialize on this as needed. We can have a default implementation that flattens this, and languages like MDL and OSL can override as needed.

Regardless of performance, I like the way a generated shader's parameterization will map 1-1 to the MaterialX parameterization when structs are supported in the language. You can avoid the extra work of parsing parameter prefixes when interfacing with the shader.

Two alternatives I can see for the higher order structure:

1. A struct is represented as a nesting of ShaderPorts, where a ShaderPort with children will identify as a struct and the generator can specialize on this.

2. Each structs is registered as a new TypeDesc objects, holding the hierarchy. The new types can then be set as any other type on ShaderPorts and the generator can specialize on this.

For (2) our current TypeDesc system is quite rigid and I think it would need a larger rewrite to support this. I'm working on changes to this system now anyway, but it's moving in the opposite direction there TypeDesc objects will become more lightweight. I think maybe option (1) is simpler and less intrusive.

One complication to work out is how to handle struct outputs in a flattening implementation. I assume these should be flattened as well, and for connections between struct ports we need logic to map the flattened parameters to each other.

[jstone-lucasfilm]

@ld-kerley We've now merged development work on MaterialX 1.39 back to the main branch of MaterialX, in preparation for the final weeks of development on the 1.39.0 release. When you have a chance, could you retarget this pull request back to the main branch as well?

[jstone-lucasfilm]

Let's close out this pull request in favor of #1831 for now, and we can borrow ideas from this original pull request if needed in the future.