Why it failed when converting netcdf to virtual zarr in MINIO server?

[forestbat]

I followed kerchunk's tutorial to add reference to my netcdf files:

import ujson
from kerchunk.hdf import SingleHdf5ToZarr

# conf.FS is instance of S3FileSystem
with conf.FS.open('s3://forestbat-private/test.nc', 'rb') as fp:
    h5chunks = SingleHdf5ToZarr(fp, 's3://forestbat-private/test.nc')
    outf = 's3://references/test.json'
    with conf.FS.open(outf, 'wb') as f:
        f.write(ujson.dumps(h5chunks.translate()).encode())

But it crashed like this:

Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Roaming\JetBrains\IntelliJIdea2023.3\plugins\python\helpers\pydev\pydevconsole.py", line 364, in runcode
    coro = func()
           ^^^^^^
  File "<input>", line 5, in <module>
  File "C:\Users\Administrator\IdeaProjects\hydrodata\venv\Lib\site-packages\kerchunk\hdf.py", line 99, in __init__
    self._h5f = h5py.File(self.input_file, mode="r")
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\IdeaProjects\hydrodata\venv\Lib\site-packages\h5py\_hl\files.py", line 562, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Administrator\IdeaProjects\hydrodata\venv\Lib\site-packages\h5py\_hl\files.py", line 235, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5f.pyx", line 102, in h5py.h5f.open
OSError: Unable to synchronously open file (file signature not found)

So where is the problem and how to solve it? Hope for your reply.

[martindurant]

This is fixed in #424; we could do a release, of you can install directly from the repo.

[forestbat]

I have updated kerchunk to 0.2.4，however the problem still occurs：

tests\test_spec_path.py:11 (test_gen_refer)
def test_gen_refer():
        obj_path = 'grids-origin/GFS/1/2019/03/03/12/gfs20190303.t12z.nc4.0p25.f001'
>       gen_refer_and_read_zarr(obj_path, storage_option=conf.MINIO_PARAM)

test_spec_path.py:14: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\hydrodata\reader\access_fs.py:120: in gen_refer_and_read_zarr
    nc_chunks = SingleHdf5ToZarr(fpj, obj_path)
..\venv\Lib\site-packages\kerchunk\hdf.py:96: in __init__
    self._h5f = h5py.File(self.input_file, mode="r")
..\venv\Lib\site-packages\h5py\_hl\files.py:562: in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
..\venv\Lib\site-packages\h5py\_hl\files.py:235: in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
h5py\_objects.pyx:54: in h5py._objects.with_phil.wrapper
    ???
h5py\_objects.pyx:55: in h5py._objects.with_phil.wrapper
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   OSError: Unable to synchronously open file (file signature not found)

h5py\h5f.pyx:102: OSError

pip show kerchunk
Name: kerchunk
Version: 0.2.4
Summary: Functions to make reference descriptions for ReferenceFileSystem
Home-page:
Author:
Author-email: Martin Durant <[email protected]>
License: MIT
Location: C:\Users\Administrator\IdeaProjects\hydrodata\venv\Lib\site-packages
Requires: fsspec, numcodecs, numpy, ujson, zarr
Required-by:

@martindurant

[martindurant]

Can you please extract out the code that's actually getting called, showing what gen_refer_and_read_zarr is?

Since you already have a filsystems, you should verify that you can access the given path and that it looks like an HDF, something like

>>> CONF.fs.cat_file('grids-origin/GFS/1/2019/03/03/12/gfs20190303.t12z.nc4.0p25.f001', end=3) 
b"HDF