-
Notifications
You must be signed in to change notification settings - Fork 14
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This adds a function, async_stream_unzip, that's the async version of stream_unzip. It's inspired by the code and discussion in the PR at #80. However, to avoid duplication, we have a threads-based layer to call into the existing stream_unzip function, rather than have an almost identical copy but with async/await where needed in it. This is similar to how https://github.com/uktrade/stream-zip provides an async interface.
- Loading branch information
Showing
6 changed files
with
201 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
--- | ||
layout: sub-navigation | ||
order: 3 | ||
title: Async interface | ||
--- | ||
|
||
|
||
An async interface is provided via the function `async_stream_unzip`. Its usage is exactly the same as `stream_zip` except that: | ||
|
||
1. The input must be an async iterable of bytes. | ||
2. The member files are output as an async iterable of tuples. | ||
3. The data of each member file is returned as an async iterable of bytes. | ||
|
||
```python | ||
from stream_unzip import async_stream_unzip | ||
import httpx | ||
|
||
async def zipped_chunks(client): | ||
# Iterable that yields the bytes of a zip file | ||
async with client.stream('GET', 'https://www.example.com/my.zip') as r: | ||
async for chunk in r.aiter_bytes(chunk_size=65536): | ||
yield chunk | ||
|
||
async def main(): | ||
async with httpx.AsyncClient() as client: | ||
async for file_name, file_size, unzipped_chunks in async_stream_unzip( | ||
zipped_chunks(client), | ||
password=b'my-password', | ||
): | ||
async for chunk in unzipped_chunks: | ||
print(chunk) | ||
|
||
asyncio.run(main()) | ||
``` | ||
|
||
> ### Warnings | ||
> | ||
> Under the hood `async_stream_unzip` uses threads as a layer over the synchronous `stream_unzip` function. This has two consequences: | ||
> | ||
> 1. A possible performance penalty over a theoretical implementation that is pure async without threads. | ||
> | ||
> 2. The [contextvars](https://docs.python.org/3/library/contextvars.html) context available in the async iterables of files or data is a shallow copy of the context where async_stream_unzip is called from. | ||
> | ||
> This means that existing context variables are available inside the input iterable, but any changes made to the context itself from inside the iterable will not propagate out to the original context. Changes made to mutable data structures that are part of the context, for example dictionaries, will propagate out. | ||
> | ||
> This does not affect Python 3.6, because contextvars is not available. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,6 @@ | ||
--- | ||
layout: sub-navigation | ||
order: 3 | ||
order: 4 | ||
title: Exceptions | ||
--- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,6 @@ | ||
--- | ||
layout: sub-navigation | ||
order: 5 | ||
order: 6 | ||
title: How to publish a release | ||
--- | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters