Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

async internally for read_elem and write_elem #1897

Open
4 tasks
ilan-gold opened this issue Mar 7, 2025 · 1 comment
Open
4 tasks

async internally for read_elem and write_elem #1897

ilan-gold opened this issue Mar 7, 2025 · 1 comment

Comments

@ilan-gold
Copy link
Contributor

ilan-gold commented Mar 7, 2025

Please describe your wishes and possible alternatives to achieve the desired result.

After #1726 people will be able to have zarr return async compatible arrays. We can update all of our internal io to pool requests together using asyncio.gather and give us more throughput, especially for cloud-based analysis.

  • Internally use async for the registered reading classes
  • Make a public async version of read_elem
  • Make a public async version of read_lazy/read_elem_lazy
  • Make a public async version of read_zarr/read_h5ad
@ilan-gold
Copy link
Contributor Author

The main issue here is the interplay between read_dispatched and the underlying methods in methods,py which call _Reader.read_elem (thus hooking back into the callback from read_dispatched). With a single set of methods.py that are all async, the functions hook into read_elem_async which can be problematic if callback is synchronous. Thus we probably need either a really clean compat layer (not sure what this would look like, another argumet? but then you just have giant if-then blocks?) or more-or-less duplciated functions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant