Stream export payloads, so that we get a bigger quota #2058
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As per https://docs.aws.amazon.com/lambda/latest/dg/configuration-response-streaming.html:
This PR switches our code to call invoke_with_response_stream instead of plain invoke, and then adapts to the new response format.
I could not get anything similar working locally with the AWS Lambda Runtime Emulator that we are using. So, unfortunately, the branch of our code that talks to real AWS now differs even more substantially than before. But... I do not currently know of an alternative.
Our test suite, run through the emulator, is still useful: it proves that the lambda function itself works, even if it can't prove that our invocation of it will be correct in production, or that our error handling works.
For this PR, I have tested that locally by configuring my dev setup to talk directly to the real, deployed lambda function. I tested with:
So, I think we can be fairly confident that this is safe to deploy.
We should reconsider this whole setup 😄.