Support AWS OpenSearch Serverless for elasticsearch sink

[AvihaiSam]

AWS OpenSearch Serverless Service endpoint is aoss instead of es.

Please add a config to distinguish ES from AOSS, and fix sign_request call.

vector/src/sinks/elasticsearch/common.rs

Line 258 in c7a4d7b

crate::aws::sign_request("es", request, credentials_provider, region).await

I'm not sure if there are future changes needs to be done afterwards.

[davidhuie-dd]

I agree that we should support this.

[zamazan4ik]

@AvihaiSam Could you please test the fix from the branch - #16590 ? I have no ability to test it on my own right now. You need to build Vector from the provided PR and test with your setup with AWS OpenSearch Serverless.

Thanks in advance!

[AvihaiSam]

Hey @zamazan4ik,
I have actually compiled a different version a week ago and tried it.
I have noticed several things:

1. the health API is not available
2. the version API is not available (at this point, my guess is that all the /_cluster APIs are not available)
3. my sigv4-auth signature was wrong, using the assume-role auth-option (maybe my version made it fail, i'm no rust expert...)
   For the last thing - I have opened a support ticket to AWS and they are looking into it.
   Since this issue makes the cluster impossible to work with we have decided to delete the cluster due do its high costs.
   So - unfortunately, I can no longer test it.

Hope to hear from AWS and will let you know if/when I will have access to a new cluster...

[zamazan4ik]

I have actually compiled a different version a week ago and tried it.

What version did you compile? Does it have the same fixes for AOSS as from my branch?

[AvihaiSam]

I tried my own implementation here

[handlerbot]

@AvihaiSam 👋🏻 I have interest in getting vector working with OpenSearch Serverless; any objection to me taking what you've got in your tree and seeing what the current gaps are, and me getting it committed with you listed as primary author if I can get to success? Or do you have current plans or thoughts?

[AvihaiSam]

@handlerbot sure. I left it there hoping someone will find the time to work on that.

[vitalvas]

I'm so interested in this support. Do you have any progress?

[handlerbot]

I have Amazon OpenSearch Serverless support implemented, and a PR is submitted: #21676 🤞🏻

FYI to @AvihaiSam: You were almost there with your earlier branch; the part I had to hurdle was tweaking the params to the AWS Signature V4 call to have it include the x-amz-content-sha256 header: https://github.com/vectordotdev/vector/pull/21676/files#diff-2fefb6a5eb1e6ec14e4df29cc639581b56cab588af42315f89e02bc6b798b8e4R280-R288 . Thank you for your earlier work, sharing it, and letting me run with it to here! 🎉

[AvihaiSam]

I have Amazon OpenSearch Serverless support implemented, and a PR is submitted: #21676 🤞🏻

FYI to @AvihaiSam: You were almost there with your earlier branch; the part I had to hurdle was tweaking the params to the AWS Signature V4 call to have it include the x-amz-content-sha256 header: https://github.com/vectordotdev/vector/pull/21676/files#diff-2fefb6a5eb1e6ec14e4df29cc639581b56cab588af42315f89e02bc6b798b8e4R280-R288 . Thank you for your earlier work, sharing it, and letting me run with it to here! 🎉

Thanks a lot for this! I had no idea...
Great work!

[handlerbot]

@AvihaiSam You can wipe https://github.com/AvihaiSam/vector/tree/feature/opensearch-serverless now if you want to clean up!