[p5.js 2.0 RFC Proposal]: Color module rewrite

[limzykenneth]

Increasing access

A color module that behaves more consistently with more powerful feature unlocks additional opportunity to explore colors and create p5.js sketches that can exist in environment that uses a wider variety of color profiles.

Which types of changes would be made?

• Breaking change (Add-on libraries or sketches will work differently even if their code stays the same.)
• Systemic change (Many features or contributor workflows will be affected.)
• Overdue change (Modifications will be made that have been desirable for a long time.)
• Unsure (The community can help to determine the type of change.)

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build process
• Unit testing
• Internationalization
• Friendly errors
• Other (specify if possible)

What's the problem?

The current color system of p5.js has a few notable flaws:

• All colors are coerced into being 8-bit RGB values and there are no support of colors wider than the RGB color space
• Colors once created have the color mode of when they are created attached to them, ie. a color created when the sketch is in RGB color mode, will always be in RGB color mode even if the sketch's color mode is changes to HSB or something else.
• Only RGB, HSL, and HSB are supported

What's the solution?

A new color module will be implemented. The some of the requirements of this new color module are:

• Full compatibility with CSS color space support.
• Possibility for addon libraries to introduce additional color spaces.
• Accurate gamut mapping.
• API as close to existing implementation as possible.
• More intuitive handling of color values (no more color mode attached to color objects)
• Potentially be usable as a standalone module/class without the need to include p5.js core Allow p5 classes (like p5.Color, p5.Vector etc.) to be used outside of the browser (node, deno, bun etc.)? #6830

The recommended references/inspirations for how this new color module should work are CSS color specs and color.js. However it is not recommended to bundle color.js as their approach, while very comprehensive, is somewhat complex and for p5.js 2.0 we should aim for as few external dependencies as possible.

At the same time while relying on browser CSS implementation may help simplify many of the implementation, it may be worth considering the potential scenario of p5.js itself, or even just the color module being used as a standalone module, being used outside of the browser.

Pros (updated based on community comments)

• More intuitive and easy to use color API in p5.js

Cons (updated based on community comments)

• Likely to break at least some sketches
• There may be some API changes to how color object works as well

Proposal status

Under review

[davepagurek]

Does this also include display-p3 to output wide gamut color, or are we mostly considering having different spaces that all map to the default srgb? (Seems like it is not yet possible for WebGL to output display-p3 to all browsers according to MDN.)

[limzykenneth]

For the color object itself, it will try to be lossless as much as possible. For things that need different color spaces, I'm still not sure whether the conversion should happen on p5.Color's side or on the function/renderer's side.

If we do it on p5.Color's side, it can potentially reduce duplication but the functions consuming p5.Color object will need to know to request the right color space, which is a bit less than ideal for simple implementation. Perhaps a default can help?

If we do it on the function/renderer's side, it will make p5.Color's implementation and extensibility better, otherwise implementing a new color space may also mean implementing many additional conversion algorithm, unless there is a good way to generalize conversion between color spaces.

[davepagurek]

Deferring to the renderer seems like a good way to let people work in whatever color space they want, and then outputting as close as possible to that based on what feature set is available. For graceful degradation, maybe each color space we add needs to know how to convert to RGB, so a renderer can fall back to that? That way, in a scenario where the browser adds a new space but WebGL does not support it yet, it would still have a functional default behaviour.

[limzykenneth]

Thinking a bit more now, one thing about deferring to the renderer is that the renderer needs knowledge of how to handle p5.Color object or color spaces itself. We can define helper functions in p5.Color to help with common conversion but that is basically just doing the conversion in p5.Color. Otherwise implementing a renderer with specific color needs may become more difficult?

[davepagurek]

Yeah, I guess I'm working off of the assumption that renderers will likely need to know what color spaces it can output to, since renderers seem tightly tied to specific platforms that have their own limited capabilities. That can potentially be different than the working color space (kind of like how currently sketches can work with HSB in their code, but ultimately the canvas stores the colors in RGB.) It probably would be for the best if whatever we do makes it so that a renderer doesn't have to know how to map every working color space to every possible output color space.

[limzykenneth]

I probably need to think a bit more on this. Just as a scenario: someone has a sketch that uses p5.Color with say display-p3 color space to do whatever it needs to, then eventually they try to port it to an imaginary renderer that was created for the Playdate, what should happen where? (Probably an extreme example but if we can handle this, the rest should be relatively easy)

[limzykenneth]

Now that I had some time to think about it, here's what I propose.

p5.Color should have a standardized way to represent colors externally, eg. as CSS string or some kind of standardized serializable JavaScript object. Renderers can request from the p5.Color object another representation that is supported by p5.Color (WebGL requesting another color space for example) if necessary, and this default supported list will be documented so renderer implementors should have access to it and request it accordingly.

For esoteric renderer (like the hypothetical Playdate renderer), they have two options:

1. Accept the p5.Color standardized representation and do the necessary conversion internally in the renderer (likely what something like Playdate need to do to achieve the dithering effect)
2. Extend p5.Color to provide conversion into the target color space (eg. what a inkjet printer renderer will need to implement to convert color into CMYK)

Not sure I have covered 100% of possible cases here, @davepagurek if you spot any gap do highlight it.

[davepagurek]

I think that makes sense! Maybe the only lingering question I have is what that internal format looks like. It probably has to encompass all the possible inputs somehow, so for example, storing standard css RGB values could work for a lot of input color spaces, but wouldn't be able to encode the wider gamut of display-p3. A css color string could work, but we'd maybe want to pick a limited subset of CSS color spaces for that to make it easier for those special renderers to write their colour conversions.

[limzykenneth]

CSS color string have quite wide support and is about as wide a support as I would like to see p5.Color implement by default so that is one place to start. Although the flexibility and extensibility of it certainly leave some room for improvement. A potential idea is to just go with a standardize serializable JS object and provide a static function to convert it into a CSS string (2D canvas fill for example accept CSS string so would be nice to just have it) if it is a feasible conversion.

I'm also not sure what the schema will be yet for this object but it should be standardized enough that renderers can work with it without needing to know much about the actual color mode of the object. The thing is that it should not try to be a fit all solution (using RGB or otherwise to represent all color gamut), which is one thing I learnt from looking at color.js, so I would defer to referencing how color.js does things where needed.

[davepagurek]

I guess the thing I'm getting at in terms of a common format is whether we can make it so the renderer doesn't need to know how to handle every input color format if it needs to e.g. convert to dithered 1-bit color, or for WebGL, convert to rgb. If either a css string or an object representation of that same data (preserving its format) is all the renderer has access to, the renderer would be left to handle all the possible options, and keep its list of handlers updated when new ones are added. If it were something like a css color string + a more common format across color schemes, then it would be easier for renderers to fall back on the common format if they encounter a new CSS string format that they weren't programmed to handle.

Do you think something like that is feasible? The downside there would be that we'd have to do some amount of format conversion in the main library rather than pushing that responsibility onto renderer developers, which adds to the size of you're just using 2D mode.

[nickmcintyre]

I read up on the new CSS color spaces and had a few clarifying questions:

• Would the internal format based on CSS strings look something like the following?

this._cssString = "color(display-p3 0 1 0)";

• How would fallbacks work in general? I found this conversion tool and its source code as a possible inspiration.
• Since WebGL supports the Display P3 color space, would it make sense to start there for experimenting with conversions and fallbacks?

[nickmcintyre]

What are people's thoughts on the color() function? For consistency, it seems like all p5.X objects should either be created with the new operator, as in new p5.Color(), or a createX() function, as in createColor().

[davepagurek]

• How would fallbacks work in general? I found this conversion tool and its source code as a possible inspiration.

I was thinking of something like that, yep! so like maybe a standard format in a non-wide gamut as a catch-all (equivalent to srgb in this example), optionally a standard format in a wide gamut to handle wide-gamut-capable renderers without having to handle each individual color space (maybe display-p3, as per your third point?), and then the original color format (equivalent to the lch color in this example.)

The main drawback I guess is that we need to handle those conversions from each new color space, and it's potentially extra work if you have a renderer that doesn't need them. The extra work issue could maybe be dealt with by only lazily evaluating the fallbacks, but I'm not sure if there's a good way to avoid having to have a bunch of color conversion code.

[limzykenneth]

I guess the thing I'm getting at in terms of a common format is whether we can make it so the renderer doesn't need to know how to handle every input color format if it needs to e.g. convert to dithered 1-bit color, or for WebGL, convert to rgb.

A common object representation is what I am thinking at the moment. How common it is will depends, I need to review color.js and CSS color again to come up with something. p5.Color object should be able to serialize to CSS strings and do conversions between all color spaces it support out of the box. Renderers/addons can extend the range of color spaces p5.Color supports (and so can convert between) or it can request an already supported color space (eg. srgb) then do a conversion within itself (renderer/addon) to a value it can use (eg. 1-bit dithering etc).

Would the internal format based on CSS strings look something like the following?

I would imagine no as there can be additional things that would be inefficient to represent with a string, having an object somehow probably is more efficient and ergonomic. Although like color.js, color objects can be serialized into CSS string.

This gives us more possibility of having a version of p5.Color that can be used outside of the browser environment as well.

What are people's thoughts on the color() function? For consistency, it seems like all p5.X objects should either be created with the new operator, as in new p5.Color(), or a createX() function, as in createColor().

I'm thinking to retain it for backwards compatibility (both in terms of code and existing concepts).

[davepagurek]

Btw @limzykenneth not sure if this changes anything, but here's another alternative to color.js that was just released: https://twitter.com/mattdesl/status/1819405253877362744

[limzykenneth]

@davepagurek It's definitely interesting and I'll look into it, although from what I can see from a glance, it does not support CIE spaces (by design) and as such don't have CIE Lab.

Another thing is that with the optimizations required for speed and size, it doesn't seem to be easily extensible so if we were to add any new color spaces that we need, it can be difficult if not impossible.

I'll probably try to put something together as it looks promising overall.

[limzykenneth]

Some quick tests seems to point to texel color adding about 10kb to the dependencies (only supporting sRGB, OKHSL, OKHSV, additional ones probably won't add too much) which is a pretty good start.

One immediate downside is that it does not have color.js interpolation functionality, which is quite sophisticated in that you can even choose in which color space to do the interpolation and have utilities to make creating gradients and stepped interpolation that was in another proposal easier. We'll likely need to implement this ourselves.

Serialization and deserialization are also a bit less than ideal and not as flexible but likely still usable. Edit: deserialization does not support color words (eg. "green", "blue") and has a lot of portions of CSS string not supported (eg.rgb(0, 128, 255) works but rgb(0 128 255) does not. Edit: and alpha deserialization also don't work

Another more "stretch goal" item is color contrast checker #6971, color.js has the functionality built in while with texel color, we will need to implement it ourselves.