Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(elasticsearch sink): Support Amazon OpenSearch Serverless #21676

Merged
merged 27 commits into from
Nov 11, 2024

Conversation

handlerbot
Copy link
Contributor

@handlerbot handlerbot commented Nov 1, 2024

Summary

The elasticsearch sink can now write to Amazon OpenSearch Serverless collections by signing requests with AWS Signature V4 algorithm, specifying the aoss service.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

How did you test this PR?

Stood up an Amazon OpenSearch Serverless collection, configured vector to send any files in a test directory to an index in that collection, verified they were written successfully by querying the index stats and the index itself.

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the "no-changelog" label to this PR.

Checklist

References

@handlerbot handlerbot requested a review from a team as a code owner November 1, 2024 07:55
@github-actions github-actions bot added the domain: sinks Anything related to the Vector's sinks label Nov 1, 2024
@handlerbot
Copy link
Contributor Author

@AvihaiSam
Copy link

AvihaiSam commented Nov 3, 2024

Thanks for the credit! Great work!

@handlerbot handlerbot requested review from a team as code owners November 5, 2024 04:20
@github-actions github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Nov 5, 2024
@handlerbot
Copy link
Contributor Author

Docs are in; I think this one is completely ready for review! 🙇🏻

@pront
Copy link
Contributor

pront commented Nov 5, 2024

Hi @handlerbot, thank you and I will take a look shortly.

@handlerbot
Copy link
Contributor Author

handlerbot commented Nov 7, 2024

@rtrieu Sent it over to you for docs re-review. The Check Spelling test is failing because today after a check-spelling upgrade on the GitHub Actions side, Non-existent discovered in website/cue/reference/remap/expressions/path.cue is considered a no-no. This isn't a file I have touched in my work in this PR, and while I'm glad to run the auto-updater for .github/actions/spelling/expect.txt when prompted, I am not sure what to do re: path.cue, and I think that should be fixed in a separate PR? 🙇🏻

UPDATE[0]: I decided to just replace non-existent with nonexistent in path.cue, and now nothing shows up as a clear error in the log, but the test is still failing? ¯_(ツ)_/¯

UPDATE[1]: I guess it's these, which are all about extra dictionaries that can't be downloaded, or non-alpha entries in the dictionary? Those are definitely not from anything I did in this! 😹

[:x: extra-dictionary-not-found](https://docs.check-spelling.dev/Event-descriptions#extra-dictionary-not-found) | 11
[:warning: non-alpha-in-dictionary](https://docs.check-spelling.dev/Event-descriptions#non-alpha-in-dictionary) | 2

Copy link
Contributor

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @handlerbot!

@handlerbot
Copy link
Contributor Author

@pront Pushed a commit from cargo fmt, please reapprove code and workflows? 🙇🏻

@handlerbot handlerbot requested a review from pront November 7, 2024 16:51
@pront
Copy link
Contributor

pront commented Nov 7, 2024

Apologies, the spell checker was broken. Please undo the spell checker related changes, rebase on latest master and push again to get a fresh report.

@handlerbot
Copy link
Contributor Author

@pront Ready!

@handlerbot
Copy link
Contributor Author

handlerbot commented Nov 8, 2024

@pront If you're working today, could I get a quick re-approve on this? It would be awesome for our internal velocity if I could get this into the next vector release, and it feels like that might be sooner rather than later. 🤞🏻🙇🏻 🚀

@pront
Copy link
Contributor

pront commented Nov 8, 2024

@pront If you're working today, could I get a quick re-approve on this? It would be awesome for our internal velocity if I could get this into the next vector release, and it feels like that might be sooner rather than later. 🤞🏻🙇🏻 🚀

Hi @handlerbot, I am doing a final pass now. We have plenty of time till the next Vector release ;)

Copy link
Contributor

@pront pront left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I appreciate the good documentation in this PR.

@handlerbot
Copy link
Contributor Author

@pront TYVM for the compliment, and thank you for the thorough and prompt review! 🙇🏻

Do we need any additional approvals re: documentation or other, or is this ready to be merged at next working time?

@pront pront added this pull request to the merge queue Nov 11, 2024
Merged via the queue into vectordotdev:master with commit 7fb5dad Nov 11, 2024
54 checks passed
@pront
Copy link
Contributor

pront commented Nov 11, 2024

@pront TYVM for the compliment, and thank you for the thorough and prompt review! 🙇🏻

Do we need any additional approvals re: documentation or other, or is this ready to be merged at next working time?

All good, now merged.

@handlerbot
Copy link
Contributor Author

Thank you @pront and @rtrieu for the reviews, and @AvihaiSam for starting this out! 🌟

jszwedko added a commit that referenced this pull request Nov 12, 2024
The new `service_type` is only used for OpenSearch AWS authentication.

Follow-up to #21676

Signed-off-by: Jesse Szwedko <[email protected]>
github-merge-queue bot pushed a commit that referenced this pull request Nov 12, 2024
The new `service_type` is only used for OpenSearch AWS authentication.

Follow-up to #21676

Signed-off-by: Jesse Szwedko <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support AWS OpenSearch Serverless for elasticsearch sink
5 participants