Support for !ENV tag in JSON Schema

[sisp]

Currently, properly configured VS Code recognizes custom YAML tags like !ENV in mkdocs.yml, e.g.

site_url: !ENV [MKDOCS_SITE_URL]

but JSON Schema validation fails with an error like this:

Incorrect type. Expected "string".yaml-schema: Site URL

That's because the subschema for the site_url property declares a "type": "string" value type:

mkdocs-material/docs/schema.json

Lines 17 to 21 in c7f927b

	"site_url": {
	"title": "Site URL",
	"markdownDescription": "https://www.mkdocs.org/user-guide/configuration/#site_url",
	"type": "string"
	},

IMO, the DX would be better if !ENV tags did not fail validation.

I've done some research on how, e.g., GitLab CI is handling the !reference tag, but custom YAML tags are invisible to JSON Schema validation and, thus, the best we could do is extend the subschema according to the value type after the tag (see, e.g., the addition of !reference in rules).

Unfortunately, the !ENV tag supports several formats

!ENV SOME_VARIABLE
!ENV [SOME_VARIABLE]
!ENV [SOME_VARIABLE, ~]
!ENV [SOME_VARIABLE, null]

which may be ambiguous with hardcoded values in some cases, e.g. !ENV SOME_VARIABLE resolves as the string "SOME_VARIABLE" when passed to the JSON Schema validation. For this reason, it might make sense to extend the schema only for a subset of the possible formats, e.g. only for

!ENV [SOME_VARIABLE]
!ENV [SOME_VARIABLE, ~]
!ENV [SOME_VARIABLE, null]

since arrays are typically improper values of properties where environment variables are used.

This is what the schema extension for the site_url property in docs/schema.json might look like

    "site_url": {
       "title": "Site URL",
       "markdownDescription": "https://www.mkdocs.org/user-guide/configuration/#site_url",
-      "type": "string"
+      "oneOf": [
+        {
+          "type": "string"
+        },
+        {
+          "$ref": "#/$defs/env_tag"
+        }
+      ]
     },

with the following new definitions:

"$defs": {
  "env_tag": {
    "title": "!ENV tag",
    "markdownDescription": "https://www.mkdocs.org/user-guide/configuration/#environment-variables",
    "type": "array",
    "oneOf": [
      {
        "items": [
          {
            "$ref": "#/$defs/env_variable"
          }
        ],
        "minItems": 1,
        "maxItems": 1
      },
      {
        "items": [
          {
            "$ref": "#/$defs/env_variable"
          },
          {
            "$ref": "#/$defs/env_default_value"
          }
        ],
        "minItems": 2,
        "maxItems": 2
      },
      {
        "items": [
          {
            "$ref": "#/$defs/env_variable"
          },
          {
            "$ref": "#/$defs/env_variable"
          },
          {
            "$ref": "#/$defs/env_default_value"
          }
        ],
        "minItems": 3,
        "maxItems": 3
      }
    ]
  },
  "env_variable": {
    "title": "Environment variable name",
    "markdownDescription": "https://www.mkdocs.org/user-guide/configuration/#environment-variables",
    "type": "string",
    "pattern": "^[a-zA-Z_][a-zA-Z0-9_]*$"
  },
  "env_default_value": {
    "title": "Default value",
    "markdownDescription": "https://www.mkdocs.org/user-guide/configuration/#environment-variables",
    "anyOf": [
      {
        "type": ["string", "boolean", "integer", "number", "null"]
      },
      {
        "type": "string",
        "format": "date"
      }
    ]
  }
}

The !ENV tag also supports an arbitrary number of fallback environment variable names when an explicit default value is provided. Declaring this general pattern isn't possible with JSON Schema to the best of my knowledge (vice versa would be possible through additionalItems¹, but that's not what we need 🤷), but we could provide a subschema for a few fallback variables (e.g. 2 in the example above) through tuple validation.

Adding support for the !ENV tag would introduce quite a bit of noise across the schema files because it can be used in many places. But it would reduce (or even avoid) false errors in IDEs like VS Code, which are a little annoying TBH.

WDYT?

Footnotes

1. A variadic tuple [number, ...string[]] can be expressed with JSON Schema like this:

   {
     "type": "array",
     "items": [
       {
         "type": "number"
       }
     ],
     "minItems": 1,
     "additionalItems": {
       "type": "string"
     }
   }
   

   Credits to https://github.com/ajv-validator/ajv/issues/1417#issuecomment-770444277 for the inspiration. 🙇

   But we'd need something similar to [string, ...string[], (string | number | boolean | null)]. 😞 ↩

[squidfunk]

Thanks for suggesting and also for trying to assess the consequences it would impose. I agree that it would be great to have validation for !ENV, but sadly we couldn't stop there. There are other tags like !relative that would need to be accounted for as well. I'm not sure if the maintainers of MkDocs plan to add more tags, as I've also proposed to add a tag called !resolve to better account for multi-level projects. Not sure whether this will be added, but you get the gist.

As you already stated, we'd need to add the env_tag declaration to each and every declaration, at least to all properties that accept strings. However, a huge problem is that we're allowing plugin authors to register their plugin schemas with our schema, which means that those schemas are not under our control. If we would start adding !ENV to each and every tag, we would also need to find a way to share the !ENV definitions (and those for other tags), so they can be used. It would be a never ending game of whack-a-mole, and likely increase the number of issue on our repository, since Material for MkDocs a popular entrypoint into the ecosystem.

A more scalable approach might be to tell MkDocs to print the configuration that it used for building the project, similar to what tsc allows with --showConfig. This could either be implemented into MkDocs itself or into a plugin. MkDocs could emit the parsed config, since tags are resolved when YAML is parsed. One problem that comes to my mind is: what if you're referencing !python/... tags to specify functions? The resulting YAML would have instances of those functions, so we would need to find a way to either filter those keys from validation, or move this to a pre- or post-validation step. So it's definitely not simple.

Thus, for the reasons stated, we're currently not considering going to add !ENV to our schema. It's just not economical.

[sisp]

Thanks for taking the time to reply so comprehensively. I acknowledge that this problem is actually more complex because of referenced external plugin schemas. But I think there's a minor misunderstanding: The VS Code YAML extension strips the custom tags before passing the document to the JSON Schema validator. So a config like

site_url: !ENV [MKDOCS_SITE_URL]

is passed to the validator like this:

{
  "site_url": ["MKDOCS_SITE_URL"]
}

Nevertheless, extending the schema might just not be scalable as you explained.

If you follow our guidelines for setting up validation, VS Code should not complain about those tags:

[...]

I've done it, and VS Code is happy with the tags, but the JSON Schema validation fails.

[squidfunk]

I've done it, and VS Code is happy with the tags, but the JSON Schema validation fails.

Yes, it will only work for VS Code, that's true. I may have phrased my last comment a bit badly. In order to make it work with external validators, my recommendation is to explore how we can teach MkDocs to print the resolved configuration.

[sisp]

What would the resolved config with !ENV tags look like? The environment variables typically haven't been set while developing in VS Code. 🤔 When there's no default value, the value of an environment variable is unknown at that point.

[squidfunk]

I recommend reading the MkDocs documentation on !ENV on how values are resolved.

[squidfunk]

That being said: it's a good idea to set default values for all !ENV usages, especially for non-nullable values like site_name:

site_name: !ENV [SITE_NAME, 'My default site name']