Include serializer from ManifestStore in Icechunk virtual references

[maxrjones]

I think we need much more work around our handling of codecs (which would get much easier with zarr-developers/zarr-python#3276), but this is a more immediate fix to support parsers that specify any serializer other than the default (e.g., big-endian bytes).

This makes your NetCDF3 example work @rsignell

• Closes ArrayBytesCodec not included in Icechunk serialization #762
• Tests added
• Tests passing
• Full type hint coverage
• Changes are documented in docs/releases.rst
• New functions/methods are listed in api.rst
• New functionality has documentation

[rsignell]

@maxrjones I ran my workflow and still got gibberish: https://nbviewer.org/gist/rsignell/32f563cfbe83c20ac01aba00190e5912
Did I need to update some other stuff in addition to virtualizarr?
Or perhaps more likely I did something else wrong. 😕

[maxrjones]

@maxrjones I ran my workflow and still got gibberish: nbviewer.org/gist/rsignell/32f563cfbe83c20ac01aba00190e5912 Did I need to update some other stuff in addition to virtualizarr? Or perhaps more likely I did something else wrong. 😕

Yeah, this fix is only for Icechunk. Kerchunk uses Zarr specification 2, which will require separate fix or adapting our Kerchunk writer to use Zarr format 3. I'd be tempted to go with the second option because converting from Zarr V3 to V2 for Kerchunk adds a lot of surface area for potential bugs, but that would be a separate PR.

[rsignell]

@maxrjones cool, I wanted to use icechunk anyway! And as you said, it does solve my use case:
https://nbviewer.org/gist/rsignell/480cf6ac5142ce2b828199bcc601a93e

[TomNicholas]

So to check my understanding, the reason this failed was because the default array->bytes codec (i.e. the "serializer") in zarr v3 terms was for little endian? Which is fine until you use it to decode big-endian data, when it will give you gibberish?

[maxrjones]

So to check my understanding, the reason this failed was because the default array->bytes codec (i.e. the "serializer") in zarr v3 terms was for little endian? Which is fine until you use it to decode big-endian data, when it will give you gibberish?

That's right.

We still likely have bugs because a user can define their own codec that doesn't subclass from Zarr python. In that case, their custom codec will be dropped by

VirtualiZarr/virtualizarr/codecs.py

Lines 49 to 63 in f3149d6

	def extract_codecs(
	codecs: CodecPipeline,
	) -> DeconstructedCodecPipeline:
	"""Extracts various codec types."""
	arrayarray_codecs: tuple[ArrayArrayCodec, ...] = ()
	arraybytes_codec: ArrayBytesCodec | None = None
	bytesbytes_codecs: tuple[BytesBytesCodec, ...] = ()
	for codec in codecs:
	if isinstance(codec, ArrayArrayCodec):
	arrayarray_codecs += (codec,)
	if isinstance(codec, ArrayBytesCodec):
	arraybytes_codec = codec
	if isinstance(codec, BytesBytesCodec):
	bytesbytes_codecs += (codec,)
	return (arrayarray_codecs, arraybytes_codec, bytesbytes_codecs)

if they use create_v3_array_metadata. This may be the cause of #770.

I expect a lot of our codec handling will get much easier with @d-v-b's refactor in Zarr-Python, which is why I have been proposing small patch fixes rather than more fundamental changes.

[TomNicholas]

their custom codec will be dropped by

Could we at least raise in that scenario?

[TomNicholas]

Add a changelog entry and this is good to go, thank you!

[maxrjones]

their custom codec will be dropped by

Could we at least raise in that scenario?

yeah we definitely should raise in that scenario.