Proposal: Semantic color aliases (both official and theme defined)

[joshgoebel]

TLDR: Many themes are impossible to represent in Base16 because we insist on binding multiple semantic meanings into singular colors: IE, "variables AND diff deleted MUST always be the same color". It's baked right into the spec. 🙁 You'll never find a Base16 theme where that's not true, but you'll find many themes in the larger world where that's not true.

This results in many themes ported to Base16 being broken or becoming "hollow shells" of what they intended to be. I think this is a bad outcome and we should be more flexible. Look at the Nord example at the top of #10... the Base16 Nord on the left honestly doesn't even look like Nord at all.

---

I think this deserves it's own topic though it's an offshoot of #10... I've quickly come to believe than many themes coming into Base16 from the "outside" can't possibly be properly represented in Base16 because of Base16's insistence on "semantic binding"... ie binding multiple semantic meanings into a single base16 color:

• "a variable MUST be the same color as Diff Deleted"
• "and also the same color as an XML tag"
• "and also the same color as..."

It's a bit larger problem than "too much red"... We don't have to look any further than the Nord examples given to see how fast this falls completely apart:

• https://github.com/chriskempson/base16/issues/162
• The "red" colour prominence in base16 editor colourschemes for non-error code #10

Nord is very clear about the intention of it's palette (emphasis mine):

Nord4: (a very light grey) For dark ambiance designs, it is used for UI elements like the text editor caret.
In the context of syntax highlighting it is used as text color for variables, constants, attributes and fields.

Nord11: (a slightly subdued red) Used for UI elements that are rendering error states like linter markers and the highlighting of Git diff deletions. In the context of syntax highlighting it is used to override the highlighting of syntax elements that are detected as errors

So immediately Nord is impossible to properly represent in Base 16 because it's a violation of the "semantic binding" of base08.

base08 - Variables, XML Tags, Markup Link Text, Markup Lists, Diff Deleted

---

Proposal

I propose we entirely unbind the semantics with aliasing... and that all the current semantic labels become aliases. We many also need aliases for mapping colors into terminal color space... (if the desire is to TRY and match default terminal colors at all).

scheme: "Nord"
# our own user defined aliases for convenience
nord0: "#2E3440"
nord1: "#3B4252"
nord2: "#434C5E"
nord3: "#4C566A"
# now the "official" semantic alias
diff_added: nord14
diff_deleted: nord11
variable: nord4
error: nord11
# ... and finally, required for compability...
base00: nord0
base01: nord2
base02: nord4
base03: nord3
# terminal colors often have fixed meanings
term09: nord11  # terminal color 9 is bright red, etc...

For backwards compatibility all the "official" alaises could map to their "semanticly bound" colors, as they do now... so if you didn't specify error or diff_deleted or variable they would all point to base08 (redish in many themes).

This would certainly require the aid/assistance/buy in of the template maintainers (to benefit from this improvement), but I think they'd welcome it... I'll speak for myself as the maintainer of Highlight.js template that I'd love to see this - and Nord is just one example of why.

Originally posted by @joshgoebel in #10 (comment)

[joshgoebel]

For builders I think the changes would be pretty simple:

• Simply allow/welcome the additional YAML keys
  • (we might consider name-spacing or nesting for the official ones)
• enforce maximum of 16 literal hex colors (we're still base16)
• Require base00 thru base15 (for compatibility)
• a color may be either a literal hex color or an alias to another alias or literal
• For sanity check (typos, etc) make sure every label is referenced at least once (ie nord44 would be an instant error)

[forrestli74]

I think the basic idea is pretty sound. I assume only base 0-15 is required. Since we want to make sure any template works for any scheme (The whole point of base16), we probably want to define a superset for all possible name aliases and all aliases should default to a base color. If what I'm saying is ok, aliases like nord4 should be ignored by templates.

Another lower level question is what syntax we want to use. An alternative syntax is to use yaml anchor

I think it's more expressive and easier for builders to implement. It even opens the door for having more than 16 colors.

[joshgoebel]

aliases like nord4 should be ignored by templates.

Yes that's the idea. They'd only exist as a convenience for scheme authors. I think in many other cases one might use the baseXX aliases to define the semantic aliases or else define named color aliases like red, green, etc.

[JamyGolden]

In #10 although I did title it The "red" colour prominence in base16 editor colourschemes for non-error code, I suggested that I suspect the problem was systemic; and the goal of the issue was to create discussion around that (most probably) systemic problem to find a path forward, so I'll close that ticket now since it achieved the goal.

@joshgoebel I like this proposal. I suggest also including diff_modified to go with diff_added/diff_deleted.

[joshgoebel]

I suggest also including diff_modified

Sure. I purposely wasn't being exhaustive... obviously an official list would need to be created if we are to go this direction... and it might involve MORE than what we currently list in our styles doc, but those we list there (including Diff Change) should certainly be included I'd imagine.

[joshgoebel]

I'd say:

• first get buy-in on the whole proposal at a high-level
• create an official list
• run that list by template maintainers (who will likely suggest additions)
• come up with an initial master list for v0.11 of the spec

We obviously will have to decide how general/specific we want to be since some scopes in some editors get crazy like: variable.inside_comment.constant.builtin (entirely made up, but I've seen scopes go deep)...

Looking at some popular editors and the most popular scopes/contexts to theme might help us insure we have a good working list. (i mean what we have now isn't terrible, but it's probably missing a few items at least)

[forrestli74]

By the way, base9 has something similar which could use as a guide for designing a list of possible aliases:
https://github.com/base9-theme/base9-core/blob/main/src/semantic.json

[actionless]

Proposal
I propose we entirely unbind the semantics with aliasing...

such en-complication of format would require to create a simple linting tool based on some of the builders, to run it on CI, which would check what all those references are resolvable and provide as a result at least minimal base16 scheme names

also i think it would be better to prefix named color references with some special characters to avoid confusing situation with old-style colors which are not prefixed with '#' and not to break backward-compatibility with old schemes by disabling colors without '#' prefix

[joshgoebel]

require to create a simple linting tool based on some of the builders

I'm not sure why it wouldn't just be the builder... CI:

• select a template that exercises all spec features
• builder builds all schemes (and reports errors)

We get it for free as soon as the builder properly supports these checks for building (which it must to do it's job).

also i think it would be better to prefix named color references with some special characters to avoid confusing situation with old-style colors which are not prefixed with '#'

An example here might help, I'm not sure I follow what you're suggesting fully.

[actionless]

An example here might help, I'm not sure I follow what you're suggesting fully.

ffffff is it "variable" name or color?

[joshgoebel]

ffffff is it "variable" name or color?

Well it could technically be either... if there was an ffffff variable, it's a variable, otherwise it's a color - that was how I imagine the scheme would work by default - any aliases that can't be resolved further are treated as colors. I'm not sure this presents a problem in the real world usage unless you can come up with a more compelling example.

I could of course invent ficticous examples, with my Bace scheme:

bace00: ...
bace01: ...

---

But we could probably also disallow variables with 6 digit hex names, and then we avoid this edge case entirely. Though I don't think a lot of people were likely to run into it in the first place.

Or force someone to add # to the hex codes where they'd be ambiguous...

# no ambiguity now
bace00: #bace01
bace01: #bace00

I'm kind of sad we made it optional personally. If that's what you were originally suggesting that's not so bad.

[actionless]

i was lazy exercising myself to come up with a 6-letter word containing only letters a-f, but great what you found better example yourself

i think it would be better to just prefix variables with smth like @ or other character which doesnt require escaping in yml, to avoid spec-ing and implementing in the builders additional resolving algorithm for deciding whatever some identifier is color or "variable"

Or force someone to add # to the hex codes

that's not worth of breaking backward-compatibility

[joshgoebel]

that's not worth of breaking backward-compatibility

It wouldn't since no aliases exist yet, there are no naming conflicts yet. The # would only be necessarily in the case of ambiguity - which is going to be RARE.

i think it would be better to just prefix variables

You mean on the value side, like this?

# reminder, # is optional 
bace00: "#bace01"
bace01: "#bace00"
bace02: "@bace00"
bace03: "@bace01"

Someone suggested the < YAML back-reference feature but honestly if we can stick with super-simple YAML I think that's preferable for the widest inter-op.

[actionless]

i meant what totally disabling colors without # would break backward-compatibility

adding resolution algorithm to determine whatever some unprefixed identifier is color or "variable" - won't break backward-compatibility but will en-complicate both spec and builder logic

[joshgoebel]

totally disabling colors without # would break backward-compatibility

I was never suggesting that.

adding resolution algorithm ... builder logic

It's quite trivial I assure you... few lines. But even if it was slightly harder - I think the main focus should be on usability for theme and template authors, not the backend implementation details.

will en-complicate both spec and

And could be explained in one or two sentences tops (and perhaps an example or two). It might be it's "too magical"... but I'll let others weigh in and decide... Using a prefix is "ok"... just I think not necessary in 99.9999999% of cases.

[actionless]

It's quite trivial I assure you... few lines.

it's ok when updating existing builder, but every of those "two lines" here and there would be a PITA if implementing builder from scratch. especially as it could be avoided without loosing the same functionality

[actionless]

i think it's also would be useful to do a research which of currently available yaml parsers do support < - if it's wide enough across different languages - it would be better decision than implementing that at builder at all

[joshgoebel]

but every of those "two lines" here and there would be a PITA if implementing builder from scratch

I've written such things from scratch before, it's simply not that complex. Truly. 💻

[joshgoebel]

@belak Kind of waiting for your high-level thoughts before I whip up a quick proof of concept and fix Nord to see how all this might work in practice. :)

[actionless]

but why to do it at all if same problem could be solved without doing it?

your motivation also totally not correlating with your previous decision to remove template-index from spec because "it was too complicated"

but writing same two lines for allowing ambiguous variable identifiers instead of specific ones is not, is "easy, not complex"

[joshgoebel]

but why to do it at all if same problem could be solved without doing it?

It has to be done one way or the other - which requires some amount of complexity, I just think my way might be a bit more elegant... sometimes a bit of magic is nice.

your previous decision to remove template-index

Not my decision... and entirely different. Each thing must be judged on it's own merits. I think aliases are a feature that would benefit MOST syntax template authors where-as I think "build the world" benefits a very, very small number of users.

allowing ambiguous variable identifiers i

I don't allow them, they (in the very rare chance they are encounted in the wild) they throw a clear error. :)

[joshgoebel]

assert.almostEqual(2, 30)

😄 True (in absolute terms) - and perhaps I shouldn't have said "two lines of code" or whatever I said... BUT... I'm always thinking in terms of top-down design, abstraction, and well hidden complexity... and from that perspective it is literally a 1 line patch that really touches a single location in the codebase (other than a few constants) and adds only a single simple and easy to understand function.

    const hexDefinitions = Object.keys(data).reduce((accumulator, key) => {
+      resolveAliases(key, data);

You can't get a lot better than that for the MAJOR new functionality this provides.

[davidscotson]

A possibly relevant area to compare with is the recently standardized forced-colors spec in browsers.

It tries to to standardize some basic semantic names that are under the control of the user (and/or the user's OS, e.g. dark mode may be activated at sundown). So you as a user can specify white on black, black on white, high-contrast, low-contrast (many of the examples talk about high-contrast but it's important to give the user control over this) or really anything to suit your own needs and any site that tries to follow the spec will do it's best to honor the users wishes.

https://drafts.csswg.org/css-color/#valdef-system-color-canvas

It hits lots of the same issues, if you don't know which colors are used (since those are under user control), how can you know the output will be readable. They suggest mostly setting the colors in foreground/background pairs, but also expect certain colors to be readable against certain other colors (e.g. Link on Canvas) rather than always setting a pair of colors.

An accessabile base16 theme that picks these up as well for consistency would be neat too, but referring to it mostly as a source of similar ideas.

[joshgoebel]

That sounds a bit like perhaps phase 2... I'm more focused on the technical details of creating the aliasing system to allow semantic color names at all... at that point there are 100 different directions we COULD go that would require lots of discussion.

That's why I suggested our initial pass might just start with the original semantic scopes from the spec.. I'll go ahead and write them out:

default_bg
lighter_bg
selection_bg
comments
invisible
line_highlight
dark_fg
light_fg
light_bg
search_text_bg
variable
xml_tag
markup_link_text
markup_link_url
markup_list
markup_code
markup_italic
markup_bold
markup_quoted
integer
boolean
constant
xml_attribute
string
class_name
inherited_class_name
function_name
method_name
diff_inserted
diff_changed
diff_deleted
support_name
regular_expression
escape_chars
attribute_id
heading
keyword
storage
selector
deprecated
embed_tag

From the official list

[belak]

@joshgoebel

@belak Kind of waiting for your high-level thoughts before I whip up a quick proof of concept and fix Nord to see how all this might work in practice. :)

I'll try to take a closer look this weekend - but after a quick glance I like the general idea, though I worry a bit about the potential complexity.

I think one other limitation is that it won't map cleanly to the 16 colors any more and therefore will only really work with GUI applications.

[joshgoebel]

worry a bit about the potential complexity.

Complexity for implementors? More choices can mean more complexity, sure. But look at the Vim template, it's already 413 lines... for many editors (Sublime, VS Code, Vim, Emacs) the amount of complexity doesn't go up - we just get higher fidelity and more diverse themes.

If the spec mandates that the baseXX palette itself still must "make sense" for some definition of "make sense" then templates could do as much or as little work as they want to support more "premium" experiences... and you can still always get a "base experience" comparable to what we already have today. (though I think #10 raises questions of how great that base experience sometimes is)

I think we also need to realize not all designers have the same goals...

I think one other limitation is that it won't map cleanly to the 16 colors any more and therefore will only really work with GUI applications.

Why would this have an impact on which applications can work with Base16? We map those semantic meanings into 16 colors now... This just allows someone to map the semantic meaning of the 16 colors differently. Or are you merely saying you're afraid that specific themes will become "less general purpose" and more targeted towards specific goals/users/applications... in which case I'd ask:

Is a designer wanting to craft amazing Base16 themes focused largely on the Terminal welcome in our community? I think the answer should be yes and that:

• not all themes will be great Terminal themes
• not all themes will be great coding themes
• not all themes will be great "general purpose" themes

Now I'm thinking there is a whole 3rd discussion here on the meta level about thus topic, the idea of a "general purpose" theme. That maybe the part of Base16 I take issue with.

• I don't think our "magic sauce" of attaching Diff Deleted to Variable has panned out.
• I'm not sure forcing this tight conformity on theme designers is a net good.

[joshgoebel]

is that it won't map cleanly to the 16 colors any more

I'm not sure I understand what you mean by this... which "16 colors" are we mapping to again? Surely you don't mean colors/hues? Were you merely trying to say the 16 palette slots that we attach semantic meaning to? I mean in that regard it'd be hard not to agree... the part of the spec that says base0B is ONLY ever fox X, Y, and Z would have to change.

Nord (and many other themes) true intents are broken in Base16 until that part of the spec starts to open up.

I'm not sure that means we need to open up ALL the slots fully... that would be another discussion... I'd personally lean more towards "yes" though and find a way to guide the resulting creativity instead of restraining it. Having 0 and 5 for background and foreground (legacy) and then letting people build on top of that sounds nice.

[belak]

is that it won't map cleanly to the 16 colors any more

I'm not sure I understand what you mean by this... which "16 colors" are we mapping to again? Surely you don't mean colors/hues? Were you merely trying to say the 16 palette slots that we attach semantic meaning to? I mean in that regard it'd be hard not to agree... the part of the spec that says base0B is ONLY ever fox X, Y, and Z would have to change.

Maybe I was misunderstanding (like I said I just did a quick pass so far), but I thought part of this proposal was allowing additional colors to be assigned in addition to assigning semantic aliases. The additional colors would mean more than 16 which wouldn't be possible in all terminals. If you're just taking about aliases, that's probably easier.

[joshgoebel]

but I thought part of this proposal was allowing additional colors to be assigned in addition to assigning semantic aliases.

Nope. I'm happy with the 16 colors (it's in our brand!), though I think we should consider shaders in the future, but that's a separate discussion. This proposal is only about making it easier for people to deploy those 16 colors how they choose, without having to fight with Base16. And making it easier to write themes that have named palettes... I think my Nord theme is much more readable with just the nord0-nord15 aliases and no other changes.

But at the end of the day there are still only 16 colors to choose from. If someone (with the power of aliases) adds 19 colors, then the builder would throw an error and say "only 16 colors, please".

[joshgoebel]

@belak Pointed out that we also need the ability to turn semantic names into ANSI (0-16) colors for terminal sake... that a template might NOT need need the hexcolor, it might need to know "which of the 16 terminal colors is the right one"... example:

Take line 248:

call <sid>hi("Comment",      s:gui03, "", s:cterm03, "", "", "")

This wants "terminal color" (0-16)... not "hexcolor". So higher up in the template:

let s:cterm00        = "00"
let g:base16_cterm00 = "00"

This would need to be replace with something like:

let s:comment_term00        = "{{comments-to16xx}}"
let g:base16_comment_term00 = "{{comments-to16xx}}"

# then later
call <sid>hi("Comment",      s:gui03, "", s:comment_term03, "", "", "")

Which would figure out which color "comments" resolves to but then output the ansiXX number code rather than the hexcode. And this is only relevant for 16-color terminals and their associated apps. Ugh. :)

[joshgoebel]

#47 is my first draft at adding "named slots" (my new name for ALL the labels) this to the spec. Still early.

[joshgoebel]

Perhaps $ for custom named slots? (to differentiate from built-in named slots)

string: "constant"
constant: "base05"
base05: $red
$red: "ff0000"

[MultisampledNight]

For context: I create schemes, originally only for Vim, but that then expanded on writing out every config for every program. The key takeaway of base16 for me is write once, build, have ready configs for everything.

The idea with the aliases sounds great to me and reminds me of Vim's highlight links, where you can easily say with

hi! link Float Number

that Float should "import" all color settings of Number. As I understood, aliases are basically that, and for the theoretical base16 version supporting that this might just be float: "number".

Now, for the actual set of those aliases: I think they should first form a reasonable hierarchy (float should link to number should link to literal), which then might link to the "default" baseXX colors (like literal should link to base09). If this all happens in a single YAML file provided by the spec and maybe in a few templates which add new ones, there's always a place to look up those defaults. Maybe builders could also provide some way to resolve those keys on request.

I do want to express concern about the differentiation strategy between color and alias though. cafe00 interpreted as color and not as alias is a bit unfortunate and could easily cause confusion, ideally the builder would just throw a hard error. This leads me to my next point: If YAML's already extended with using $ for custom named aliases, why even enforce the string thing? Strings could be interpreted as colors with no exception, if a text has no quotes around it, it could be unambiguously interpreted as alias. I don't see a reason yet to differentiate official and usermade aliases either.

[joshgoebel]

I do want to express concern about the differentiation strategy between color and alias though. cafe00 interpreted as color and not as alias is a bit unfortunate and could easily cause confusion

This has already been addressed - there are several solutions... The build tool can throw an error if a lookup is ambiguous... or we can mandate the use of # for color, or we could use namespaces... but this is a very tiny edge case in my book. The $ naming for variables also solves this.

If YAML's already extended with using $

It's not extended, $ is just one of the characters YAML supports as a key name - and happen to be used in may languages as a variable identifier - so it would be familiar to some.

if a text has no quotes around it, it could be unambiguously interpreted as alias.

That isn't something we know, that's just how the YAML is written - after it's parsed there are no quotes anywhere, only raw strings.

I don't see a reason yet to differentiate official and usermade aliases either.

It's to avoid future overlap and conflicts between system slots/alises and user variables.

[joshgoebel]

I think they should first form a reasonable hierarchy

That's also a big leap in complexity (with lots of edge cases) but it's technically doable. With even a single tier though we're 30x more flexible than base16.

[joshgoebel]

I've create a new repo for what I'm calling the Base17 style guide: https://github.com/base16-project/base17

I'd love feedback, the spec itself can be found under style_guide.md.

In my opinion this is how we move forward - now that Chris has reasserted his ownership of Base16.

• We leave the Base16 style spec alone - if our tools work with it (and I think they should) great, but we shouldn't be changing it.
• We create a new forwards compatible spec, Base17. All Base16 schemes are valid Base17 schemes - just base17 allows you to do MORE.
• We rename base16-schemes to base17-schemes and then we can begin to evolve the base17 style system (that we own) - or introduce additional style systems (Ansi16, etc)

I think getting our tooling to accept and work with the fact that there are real and divergent (the input file won't all look the same) style systems is the first step to making all of this more concrete.

[belak]

I would recommend separating semantic color aliases from color variables. Having 2 separate proposals will make it easier to talk about and tweak each of the ideas separately and see their value individually.