Make Godot GLSL shader files interoperable with standard GLSL tools

[danderson]

Note: I am volunteering to ship this myself, if this proposal is acceptable to Godot devs. I have a working prototype already, and I volunteer to get it merged and update documentation.

Describe the project you are working on

I'm experimenting with procedural generation algorithms. For many of them I use compute shaders to speed up the generation process.

Describe the problem or limitation you are having in your project

Godot uses a non-standard GLSL format. Conceptually, the format stores several independent shaders in a single file, separated by #[shader_stage_name] markers (one of vertex, fragment, tesselation_control, tesselation_evaluation, compute). A special #[versions] section also exists, which allows the user to define "variants" of shader with C preprocessor directives. This format is used internally by Godot renderers for the shader code they need, and it's also the user facing format for compute shaders. In the case of compute shaders, the format must be obeyed but only #[compute] and #[versions] sections can appear in the file.

This format is specific to Godot, which means that standard GLSL tools don't function well on Godot shader files. Take for example the trivial compute shader from the manual:

#[compute]
#version 450

// Invocations in the (x, y, z) dimension
layout(local_size_x = 2, local_size_y = 1, local_size_z = 1) in;

// A binding to the buffer we create in our script
layout(set = 0, binding = 0, std430) restrict buffer MyDataBuffer {
    float data[];
}
my_data_buffer;

// The code we want to execute in each invocation
void main() {
    // gl_GlobalInvocationID.x uniquely identifies this invocation across all work groups
    my_data_buffer.data[gl_GlobalInvocationID.x] *= 2.0;
}

If you look at the GLSL specification, this file is invalid in 2 ways:

• #[compute] is not a valid GLSL preprocessor directive. Section 3.3 of the spec lists the valid directives, and explicitly says that directives not listed result in a compiler error. Empirically, all GLSL tools (linters, IDE extensions, compilers) I've tried do indeed choke on this directive.
• #[compute] must come before #version, because the Godot format requires all code to be inside a section. However, section 3.3 of the spec says that the #version directive must appear before anything else, except comments and empty lines.

This only gets worse if you add a #[versions] section, because the contents of the versions section is not valid GLSL syntax at all.

This makes it hard to work with compute shaders in Godot (and, I presume, also rendering shaders, but I don't know). Basic syntax highlighting works, but other IDE functionality is degraded: there is no intelligent complete/navigation because GLSL language servers can't parse the file, there is a constant sea of red underlines from the IDE trying and failing to parse things, and so on. It's not a fatal problem, but it makes shader hacking in Godot quite unpleasant.

This also causes an adoption problem: you can't just grab some shader code off the internet, or from a GLSL shader library, and drop it in your Godot project. Again this is not a huge barrier since you usually just have to add #[compute] at the top, but it's a less smooth experience than in other engines. It also forces new developers to learn GLSL without good IDE support, if they want to use Godot.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The discussion in #10880 sketches a solution to this (which is also detailed below) : make "Godot-style GLSL" compatible with regular GLSL. This means replacing the #[section] marker syntax with a combination of compiler #pragma directives and preprocessor #ifdefs. The result is a shader file that is standard GLSL, with additional Godot-specific hints to support multi-stage and multi-version shaders.

In addition (although this can be separated into a different proposal), add support for specifying the type of simple shaders in the filename, e.g. myshader.compute.glsl gets compiled as a compute shader with no extra work needed.

Obviously, we cannot break compatibility with existing Godot GLSL shaders. Fortunately, this proposal is completely backwards compatible: all existing shader code keeps working exactly the same, the shader loading code can trivially detect the format per-file and adapt.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I got carried away a little bit while prototyping, and implemented this proposal already 😅 , if you prefer to just read code: godotengine/godot@master...danderson:godot:push-xvurqvqkplyr . I broke down the change into small incremental commits, so I recommend reading one commit at a time, for smaller diffs. Please consider this just a prototype to show that it can be done with low pain, and to try the proposed format interactively. I'm happy to adjust the implementation based on feedback, of course!

The new logic is entirely in RDShaderFile, which implements the current Godot-style GLSL format. Conceptually, RDShaderFile acts as a pre-preprocessor stage that figures out what shader stages and variants are present, assembles the correct source code for each one, and invokes the RD's SPIR-V compiler for each.

I added a second parsing codepath in RDShaderFile, which processes the new format (described below). The top-level parse_versions_from_text method uses a quick heuristic to figure out which of the 2 formats it's looking at, and delegates to either the existing format parser, or the new one. Additionally I factored out a few common helpers that both parsers use, to reduce duplication and improve readability.

Everything else stays the same. Aside from a tiny API expansion to support filename-based stage identification in the editor, the rest of Godot is unaware of the new format. The editor resource for GLSL shaders just hands the source text to RDShaderFile, and gets back a pile of compiled SPIR-V and possibly compiler errors. Similarly the GLSL compiler itself doesn't need any changes, it just receives slightly different source text from RDShaderFile.

New file format

The new format is standard, spec-compliant GLSL. Two compiler pragmas tell Godot the desired shader stages and variants. For each {stage, variant}, the entire file is sent to the GLSL compiler, with two additional preprocessor #defines that specify the stage and variant. The shader code can then use standard #ifdefs to specialize the code as it likes.

Here is the compute shader from the Godot manual in this new format (compare to old one shown earlier):

#version 450

#pragma godot_shader_stages(compute)

// Invocations in the (x, y, z) dimension
layout(local_size_x = 2, local_size_y = 1, local_size_z = 1) in;

// A binding to the buffer we create in our script
layout(set = 0, binding = 0, std430) restrict buffer MyDataBuffer {
    float data[];
}
my_data_buffer;

// The code we want to execute in each invocation
void main() {
    // gl_GlobalInvocationID.x uniquely identifies this invocation across all work groups
    my_data_buffer.data[gl_GlobalInvocationID.x] *= 2.0;
}

This shader is trivial since it only has one stage and no variants, so all that's required is to declare the desired stage with the godot_shader_stages pragma. Alternatively, if the file is saved as blahblah.compute.glsl, the stage is deduced from the file extension, and the #pragma is not required.

Here's a skeleton of a more complex shader file, which has both vertex and fragment shaders as well as 2 versions:

#version 450

// This file contains both vertex and fragment shaders.
// The user also wants a 2nd variant with more expensive raymarch processing
#pragma godot_shader_stages(vertex, fragment)
#pragma godot_shader_versions(standard, raymarching)

// Common parts shared by all stages/variants
#include "common_defs.glsl"

struct Vertex {
    vec3 position;
    vec3 normal;
    vec2 uv;
};

// Vertex shader specific code. Godot defines this macro when compiling the vertex stage only
#ifdef GODOT_STAGE_VERTEX

// Vertex specific layouts/etc. here

void main() {
    do_common();

  // Code for standard variant only
#ifdef GODOT_VERSION_STANDARD
    fast_vertex_process();
#endif
  // Code for raymarching variant only
#ifdef GODOT_VERSION_RAYMARCHING
    cast_lots_of_rays();
    make_super_pretty();
#endif

    cleanup_common();
}

#endif // GODOT_STAGE_VERTEX, end of vertex shader

// Fragment specific code
#ifdef GODOT_STAGE_FRAGMENT

#include "fragment_stuff.h"

void main() {
    calculate_albedo();
#ifdef GODOT_VERSION_RAYMARCHING
    // The raymarching vertex shader gave us extra info we can use
    adjust_occlusion();
#endif
}

#endif // GODOT_STAGE_FRAGMENT

Of course this is more complex, and standard GLSL tools will need some manual configuration to set the right GODOT_{STAGE,VERSION}_* macros. However, this is fairly easy to do, and is only necessary for these more complex "multi-shaders". The one-stage, one-version compute shader above can be processed by standard tools out of the box. This also leaves open the possibility of tools adding explicit Godot support: the GLSL spec says that unknown #pragmas must be ignored, but in future third party tools could learn the Godot pragmas, and do the right thing even for complex shaders. This is not required, just a possibility for the future.

For a more realistic example, I picked a random complex shader from the Godot source tree (lm_blendseams.glsl, a vertex+fragment shader with 2 variants) and ported it to this new format. You can see the result in https://gist.github.com/danderson/f5b518b66c1b7670016b3a817cbefbd5. (Note this was done on an earlier prototype with different pragma names, but the semantics are the same).

Detailed semantics

These details are easy to change, as long as we stick to some requirements:

• The source file must be standard GLSL.
• It must be possible to easily disambiguate the old vs. new formats for parsing.

The parser detects the format by looking for a line that starts with #[ (not standard GLSL, use old parser) or #version (this cannot appear before #[ in the old format, use new parser).

The old parser is unchanged, except for some refactoring to extract helper functions.

The new parser scans the source file and handles the following specially:

• #version: remember for use later when compiling.
• #include: handled identically to current parser.
• #pragma: ignore non-Godot pragmas. For Godot pragmas:
  • No duplicates allowed, e.g. only one #pragma godot_shader_stages(...) per file.
  • The only two valid Godot pragmas are godot_shader_stages and godot_shader_versions. Any other godot_* pragma is a compile error (to reserve the namespace for future additions).
  • The valid values for godot_shader_stages are vertex, fragment, tessellation_control, tessellation_evaluation and compute. For backwards compatibility, we also allow tesselation (typo, with one L). Unknown values, or duplicates, are a compile error.
  • The valid values for godot_shader_versions are ASCII identifiers. No duplicates allowed, and duplicate checking is case insensitive: you can declare a version tRiAnGlEs if you like pain, but you can't have another version triangles as well.

Rules for a structurally valid shader file:

• At least 1 shader stage declared, either with #pragma godot_shader_stages or derived from the filename.
• Compute shaders cannot coexist with render shaders (I just carried this constraint over from the existing parser).
• Version/variant declaration is optional. If the file does not request versions, we compile a single default "" version just like the old parser.

If the file is structurally valid, then for each stage+version:

• Construct adjusted source file: #version header, then Godot #defines, then the rest of the source file (with any #includes inlined).
  • When compiling shader stage blah, we add #define GODOT_STAGE_BLAH.
  • When compiling version foo, we add #define GODOT_VERSION_FOO. If the user didn't ask for custom versions, we don't define GODOT_VERSION_....
• Hand this source to the RD's GLSL compiler. The GLSL compiler handles all other preprocessor directives (non-godot pragmas, #ifdef, macro expansion, etc.).
• Record the bytecode and compiler errors, exactly like the old parser.

Implementation roadmap

Assuming this proposal is acceptable and we resolve any open questions, I volunteer to ship this feature. This means:

• Polish my prototype code, send PRs, incorporate feedback, all the usual software development stuff :)
• Update the manual's compute shader section to use the new format, including: suggestion that users use an IDE with GLSL support to get nice things when editing their shaders, and leaving a side-note about the current format to say that it still works and isn't going anywhere.
• Maybe port Godot's internal GLSL files to the new format? It's trivial to do, I just don't know if renderer devs would like me to do this, or leave their shaders alone :)

Open questions/syntax war 😂

(assuming the proposal in general sounds good)

• What names do we want for the pragmas?
  • The prototype uses godot_shader_stages and godot_shader_versions to reserve a Godot-specific prefix, and match the names from the existing format. This also matches the prefix for the macro namespace in C#, for consistency.
• What names do we want for the preprocessor macros?
  • The prototype uses GODOT_STAGE_<uppercase stage name> and GODOT_VERSION_<uppercase version name>. Again reserving a prefix for Godot macros, and and matching the names to the names of the pragmas.
• If we keep the stage-from-filename logic, what file extensions to use?
  • For prototyping I made it very explicit: .vertex.glsl, .fragment.glsl, .tessellation_control.glsl, .tessellation_evaluation.glsl, .compute.glsl. I decided to keep the .glsl suffix so that the shader language is immediately obvious, and so that in future if Godot adds HLSL/WGSL/SPIR-V/whatever support, RDShaderFile doesn't have to start using horrible heuristics to guess which compiler to use. I used the same stage names that RDShaderFile uses internally, just to avoid defining yet another set of names for everything, but that means they are quite verbose, especially the tessellation shaders.
• Should we allow GLSL files that don't declare any shader stage? Currently #includes are painful because Godot tries to import them and reports errors. We could say that a .glsl file with no declared stages is "valid", we just don't invoke the SPIR-V compiler and the editor says there are no shader versions/stages available (or maybe it has UI to say "this looks like an include file, click one of these buttons to turn it into a top-level shader of type X" ?).
• Should we allow compute shaders to coexist with render shaders? The current parser doesn't allow it, but I couldn't find a reason. Maybe it would be useful as the Godot renderers get more advanced, to have a file that includes compute+vertex+fragment stages all in one? I don't know.

If this enhancement will not be used often, can it be worked around with a few lines of script?

I expect this to be used by anyone writing compute shaders for Godot projects, as well as people writing custom render shaders for advanced uses (or Godot devs hacking on the renderers).

The only workaround is what some people do today: put the actual shader code in a separate include file, and include it from the non-standard top level GLSL file. This works, but isn't very ergonomic because every time you edit the included shader the editor throws compile errors complaining that there is no #[blah] in the include file, and you end up with 2 GLSL files for even a trivial shader.

The other "status quo" workaround of course is to ignore the IDE sadness when editing your shaders, and live without autocomplete/linting/etc. Obviously this "works", but it's not a good user experience IMHO.

Is there a reason why this should be core and not an add-on in the asset library?

Using compute shaders in Godot should be a pleasant default experience without requiring external plugins/add-ons, IMHO. Enabling the use of standard GLSL source code and tools is a big enough usability/quality of life improvement to justify being in core.

The code burden is also quite low: +255 lines of code net for my prototype, and I aimed for readability/maintainability rather than "clever" compact code. The change is isolated to the internals of a single class, and the +255LoC includes full backwards compatibility with existing shaders. This seems like a small price to pay in exchange for the ability to use standard compliant GLSL source code.

Proposal changelog

• 2024-12-02: switched pragma/macro prefixes from gd_ to godot_, per Calinou's suggestion to make the prefixes match the reserved namespace for C# macros, and geekley's suggestion to avoid "GD" causing confusion with GDScript and GDShader.