Add filename hinting to multipart encoder #316
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Allow the multipart encoder to do the same filename hinting as `requests` does.
jhollowe
added a commit
to jhollowe/proxmoxer
that referenced
this pull request
Oct 2, 2021
Until requests/toolbelt#316 is merged, this will allow requests_toolbelt to send all the data needed by Proxmox
jhollowe
added a commit
to proxmoxer/proxmoxer
that referenced
this pull request
Oct 7, 2021
* Add switchover to requests_toolbelt for large file When requests handles files, it reads the entire file into memory and sends in a single payload. Currently, the python SSL tooling does not support single payloads larger than 2^31-1 bytes (https://bugs.python.org/issue42853#msg384566) This means that files ~ >= 2GiB cannot be sent using requests This commit allows a switchover to the streaming multipart payload provided by requests_toolbelt (if installed) * Add patch for requests_tollbelt filename issue * Until requests/toolbelt#316 is merged, this will allow requests_toolbelt to send all the data needed by Proxmox * Simplify large file handling * Fix compatibility issues with older python versions * Fix problem with non-JSON responses with errors * Add multi-file safety and backup upload attempt * Checks the total file size, not just the size of each file * Throws OverflowError if payload will be too large for SSL * Suggests using `requests_toolbelt` for medium-large files, but uses `requests` with the associated memory hogging * Uses `requests` for uploads under a threshold (currently 10MiB)
|
Any update to if this can be merged in? |
|
Can this be merged in? |
2 tasks
|
Sorry for the delay @jhollowe When you sent this CI was broken and I lost track of it. Can you rebase and push and add a test so this doesn't regress? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Allow the multipart encoder to do the same filename hinting as
requestsdoes.