enhancement(aws provider): Add ability to disable request signing #20973
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This can be useful when sending anonymous requests to AWS S3. Potentially it could be useful in other situations as well (e.g. sending to AWS API compatible endpoints that don't support signing). I'd really have liked to introduce this as a new "strategy" for AWS authentication configuration since it is mutually exclusive with the others, but given the way AWS authentication configuration is currently implemented as an untagged enum, adding it to the default config enum seemed like the best option. If/when we refactor this to follow https://github.com/vectordotdev/vector/blob/master/docs/specs/configuration.md#polymorphism then we can move it. Signed-off-by: Jesse Szwedko <[email protected]>
github-actions
bot
added
domain: sinks
Signed-off-by: Jesse Szwedko <[email protected]>
Datadog ReportBranch report: ✅ 0 Failed, 443 Passed, 0 Skipped, 4m 5.35s Total Time |
Signed-off-by: Jesse Szwedko <[email protected]>
github-actions
bot
added
the
domain: sources
Comment on lines
+705
to
+710
| match config.auth { | ||
| AwsAuthentication::Default { sign, .. } => { | ||
| assert!(!sign); | ||
| } | ||
| _ => panic!(), | ||
| } |
There was a problem hiding this comment.
This could probably be more concise as a assert!(matches!(...))
There was a problem hiding this comment.
Mmm, true, though I like it as-is since it matches the other tests here.
Comment on lines
+207
to
+211
| .set_http_client(Some(SharedHttpClient::new(connector))) | ||
| .set_sleep_impl(Some(SharedAsyncSleep::new(Arc::new(TokioSleep::new())))) | ||
| .set_identity_cache(Some(auth.credentials_cache().await?)) | ||
| .set_region(Some(region.clone())) | ||
| .set_retry_config(Some(retry_config.clone())) |
There was a problem hiding this comment.
Why did these all become options? Somewhat confusing since the credentials provider doesn't appear to be one.
There was a problem hiding this comment.
Ah, good question, in hindsight I should have left a comment explaining this. This change was because I wanted to use set_credentials_provider to pass in an Option (since the credentials provider can be None). This requires a &mut self, and modifies in place, rather than a self where it returns an updated self. I updated all of the calls for consistency but I think I could leave the existing calls and just use set_credentials_provider for the one where I need to pass an Option if preferred.
There was a problem hiding this comment.
The builder has <option> and set_<option> methods for all configuration options. The ones with set_ take an Option (and &mut self) where the ones without do not (and take a self).
|
Just thought of adding this extra information here in case it helps anyone / for anything .With respect to anonymous access to S3 (via S3 sink) for whose use case this unsigned request feature can be used, AWS does not allow multipart upload (for reason only known to AWS ) . S3 supports file uploads in chunks of 5 MB., and hence, only one 5 MB file can be uploaded in one go at max with unsigned request. There is no official document with this regards. Post my discussion in discord (link mentioned in issue description) , I was experimenting with batch uploads of files using AWS cli with anonymous access instead of relying on vector until this feature gets into main branch. And I just happened to stumble on this error. |
lukesteensen
approved these changes
Aug 5, 2024
|
Hey @jszwedko Even before its merged to main branch , I have started testing it 😝 .So kindly excuse my impatience. For some reason, the logs are not getting delivered to my S3 bucket with the made changes. I have detailed my experiment below. Let me know if it's a bug or if I am going wrong somewhere in my steps. Pr-requisitesThe target And the instance from where vector is being run has <MY_RESTRICTED_IP>. Steps to reproduce the issueStep -1 Building from sourceStep -2 Drafting a dummy log generator and Anonymous access based S3 sink vector configStep 3: Set the trace mode for checking out trace log entries for debuggingexport VECTOR_LOG=trace
export RUST_LOG=traceStep 4: Running vectorWhen vector is run I get (not some clear) errors which looks like below :-
I unset the trace for both RUST_LOG and VECTOR_LOG so that noise can be filtered out and I got below WARN level logs. Further, I tested with aws cli with sample file just to make sure if my permissions were correct. The sample file got uploaded successfully. The command used for it is given below :- Thought of putting these now here, so that if any issue exists, it can be corrected in the PR. |
|
@rams3sh Thanks for trying this out proactively! That's unfortunate to hear it doesn't seem to be working for you. I'll try to reproduce and see if I can figure it out this upcoming week. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This can be useful when sending anonymous requests to AWS S3. Potentially it could be useful in other situations as well (e.g. sending to AWS API compatible endpoints that don't support signing).
I'd really have liked to introduce this as a new "strategy" for AWS authentication configuration since it is mutually exclusive with the others, but given the way AWS authentication configuration is currently implemented as an untagged enum, adding it to the default config enum seemed like the best option.
If/when we refactor this to follow
https://github.com/vectordotdev/vector/blob/master/docs/specs/configuration.md#polymorphism then we can move it.
Prompted by a user in discord: https://discord.com/channels/742820443487993987/1267892632319692802/1267892632319692802