16bit png/avif decode option

[Ceeeeeeeeb]

Is your feature request related to a problem? Please describe.

I'm trying to decode a 16bit png, now I'm using upng.js which could correctly treat it as a 16bit per channel image, but stores the pixels as a bigendian Uint8Array, so I have to manually combine two 8bit value to the orignal 16bit value(this takes a lot time).

so I wonder could it be possible to decode the image as a Uint16Array directly?

Describe the solution you'd like

the decode function take an optional param (just like the avif.encode function does now) consider the bitDepth and return an Uint8Array which stores littlendian 16bit values，so I can directly convert it to Uint16Array without any manual conversion.

Describe alternatives you've considered

Consider the trade off between file size & encode/decode time cost, maybe avif.decode the better choice?

[jamsinclair]

@Ceeeeeeeeb yes, this is definitely possible. I'll need to think about this and how it fits into the project.

The packages contained in this repository are more-or-less the same codecs available in https://github.com/GoogleChromeLabs/squoosh. The project is focused on image optimisation and allowing easy interoperability between the image formats, as such only 8bit RGBA image arrays are used.

While it would not be that hard to add this functionality, it's more a question of whether we should.

[Ceeeeeeeeb]

Thanks for your reply, now I've found a workaround: simply swap the bigendian bytes then cast the Uint8Array to Uint16Array.

But I'm still hoping for 16bit image decode support for png/avif decode (I've not found other avif web decode package can do this).

Because we are developing a medical web viewer, which means higher bit depth is needed for medical diagnose. Regardless of whether this feature is ultimately supported or not, thank you for your consideration.

[jamsinclair]

@Ceeeeeeeeb it ended up being slightly more involved than I initially thought.

I've released a beta version that you can install with npm install --save @jsquash/[email protected]

If this works ok for your use cases I can look at tidying this up and creating a stable release.

The beta package contains an explicit method for decoding to 16-bit RGBA data. It works on both 8-bit and 16-bit images.
e.g.

import { decodeRgba16 } from '@jsquash/png';

const imageBuffer = await fetchYour16BitImage();

// The data array will always be an instance of Uint16Array
const { data, width, height } = await decodeRgba16(imageBuffer);

// You might want to check that the 16-bit color values are correctly mapped in the data array

You can also encode to a 16-bit PNG.

import { encode } from '@jsquash/png';

async function create16bitImage(src) {
  const pixels = new Uint16Array(4 * 50 * 50);
  for (let i = 0; i < pixels.length; i += 4) {
    pixels[i] = 0; // R
    pixels[i + 1] = 65535; // G
    pixels[i + 2] = 0; // B
    pixels[i + 3] = 65535; // A
  }
  return {
    data: pixels,
    width: 50,
    height: 50,
  };
}

const rawImageData = await create16bitImage();
const png16bitBuffer = await encode(rawImageData, { bitDepth: 16 });

Edit: I'll need to think about how Endianess plays a role. From my knowledge PNG always requires Big Endianess, according to the standard. Curious why you need the opposite?

[Ceeeeeeeeb]

@jamsinclair Thanks，I've tried the beta package and it works well for my use case.

About the endianess, I need to pass the Uint16Array to webgl and render it to a canvas.
if the data decoded remains in big endianess, I have to manually convert it either on CPU or on GPU:
swap the bits on CPU may cost some extra time; while swap the bit on GPU means special sample method & different texture size vs screen size which takes a lot more processes.

And about the decoding perfomance, I tried to compare the decode time cost between your beta package and upng.js, sadly I found that decodeRgba16 is slower than my current using work flow:

1. decode the png file with upng.js.decode
2. swap the big Endianess bytes to little Endianess
3. cast the Uint8Array to Uint16Array using the same buffer.

var png = UPNG.decode(data); 
var arr = png.data;
for (let i = 0; i < arr.length - 1; i += 2) {
    [arr[i], arr[i + 1]] = [arr[i + 1], arr[i]];
  }
var uintArray = new Uint16Array(arr.buffer);

my test image will be attached to this comment.

Futher more, is it possible to support 16bit decode for other format like avif?

[jamsinclair]

Thanks for the great feedback @Ceeeeeeeeb

And about the decoding perfomance, I tried to compare the decode time cost between your beta package and upng.js, sadly I found that decodeRgba16 is slower than my current using work flow:

On my MacOS with M1 CPU I got these results in the browser:

Upng: 27.93ms average (won 96 of 100 runs)

jSquash PNG: 35.45ms average (won 4 of 100 runs)

Overall: Upng was faster by 7.52ms (26.9% faster) on average.

You're right, I think that's to be expected. There's a bit of a performance overhead with WASM as we have to both load the module and transfer data between the two contexts. Upng seems to be well written and performance tuned as it was made for the well known photo editing software Photopea.

Futher more, is it possible to support 16bit decode for other format like avif?

With the current encoders, it seems like 12 bits is the maximum color depth available for encoding.

The libavif library that we currently use only supports, 8, 10 and 12 color depths. I see some experimental support for 16 bits, but I'm unsure on how stable that is.

• https://github.com/AOMediaCodec/libavif/blob/e3604a7f4cd088b5694757d45bb9783d429c7428/src/write.c#L1723

[jamsinclair]

It looks like the newer JPEG XL format can support 32 bit values per channel. So theoretically I think RGBA16 lossless images could efficiently be shared in that format. I'm not sure if those images can be viewed as intended or whether you would need to do your own post-processing to get the correct RGBA values.

[Ceeeeeeeeb]

ok, I have to use upng for png decoding for better performance 😔, but I will try libavif for avif files, actually we just need 10bit precision (as most medical monitor only support 10bit). Thanks a lot for your kind reply!

By the way, currently @jsquash/avif don't support choose bitdepth either, perhaps I need a decodeRgba10 interface?

[jamsinclair]

@Ceeeeeeeeb That is correct. I'll need to implement that for the avif package. I have created a proof of concept so it's definitely doable.

I'll need to think of if there's a cleaner interface for sharing the decoded data. To avoid breaking changes, for now, I'll likely proceed with new methods like:

• decodeRgba10
• decodeRgba12

Please note that the AVIF decoding speed is probably much slower than PNG. However, due to the smaller AVIF file sizes you may still slightly save on speed overall (e.g. faster image transfer on the network than PNG)