Update tutorial.rst to include section about accessing Zip Files on S3 (

zarr-developers#1615) * Update tutorial.rst to include section about accessing Zip Files on S3 Per discussion here, add information about about accessing zip files on s3: zarr-developers#1613 * Update release.rst * Implement d-v-b's suggestions --------- Co-authored-by: Davis Bennett <[email protected]> Co-authored-by: Josh Moore <[email protected]>
jhamman · Jan 16, 2024 · 1d56da0 · 1d56da0
1 parent 68c87bb
commit 1d56da0
Show file tree

Hide file tree

Showing 2 changed files with 27 additions and 0 deletions.
diff --git a/docs/release.rst b/docs/release.rst
@@ -45,6 +45,8 @@ Docs
 * Minor tweak to advanced indexing tutorial examples.
   By :user:`Ross Barnowski <rossbar>` :issue:`1550`.
 
+* Added section about accessing zip files that are on s3.
+  By :user:`Jeff Peck <jeffpeck10x>` :issue:`1613`.
 
 Maintenance
 ~~~~~~~~~~~

diff --git a/docs/tutorial.rst b/docs/tutorial.rst
@@ -1000,6 +1000,31 @@ separately from Zarr.
 
 .. _tutorial_copy:
 
+Accessing Zip Files on S3
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The built-in `ZipStore` will only work with paths on the local file-system, however
+it is also possible to access ``.zarr.zip`` data on the cloud. Here is an example of
+accessing a zipped Zarr file on s3:
+
+    >>> s3_path = "s3://path/to/my.zarr.zip"
+    >>> 
+    >>> s3 = s3fs.S3FileSystem()
+    >>> f = s3.open(s3_path)
+    >>> fs = ZipFileSystem(f, mode="r")
+    >>> store = FSMap("", fs, check=False)
+    >>> 
+    >>> # cache is optional, but may be a good idea depending on the situation
+    >>> cache = zarr.storage.LRUStoreCache(store, max_size=2**28)
+    >>> z = zarr.group(store=cache)
+
+This store can also be generated with ``fsspec``'s handler chaining, like so:
+
+    >>> store = zarr.storage.FSStore(url=f"zip::{s3_path}",  mode="r")
+
+This can be especially useful if you have a very large ``.zarr.zip`` file on s3
+and only need to access a small portion of it.
+
 Consolidating metadata
 ~~~~~~~~~~~~~~~~~~~~~~