Enhancement request - Please provide an OpenSearch destination

[ryn9]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Elastic made it latest libraries not work with OpenSearch
Please make a dedicated OpenSearch destination using the latest OpenSearch libraries.

Attempted Solutions

n/a

Proposal

Elastic made it latest libraries not work with OpenSearch
Please make a dedicated OpenSearch destination using the latest OpenSearch libraries.

References

Please have a look the blog post about client libraries available for OpenSearch: https://opensearch.org/blog/community/2021/08/community-clients/

Version

n/a

[protochron]

One way to get this going in the meantime is to use the AWS sigv4-proxy to sign the requests to OpenSearch. It's the same workaround outlined in #6204, so it's not ideal but it does work.

[zamazan4ik]

@jszwedko I suppose we need some discussion here.

OpenSearch is getting more and more popular in the community for different reasons, so I think OpenSearch is kinda important for Vector.

If we agree to support OpenSearch, the main remaining question here - how it should be implemented in Vector. I suggest create dedicated source/sink for OpenSearch, even if it will share right now a lot of codebase with an existing ElasticSearch source/sink. In the future I guess OpenSearch and ElasticSearch will diverge more and more.

What do you think?

[jszwedko]

I'm open to creating a new opensearch sink. At present, I think it could largely just wrap the elasticsearch sink and set some defaults. If the HTTP APIs between the two products diverge more, than we could split up the implementation more, but I think just wrapping would be sufficient for now.

[ryn9]

Forgive me for not knowing what's under the hood - but please know that elastic supplied libraries after 7.14 have be modified to specifically not work with OpenSearch. So if you are using these libraries - it would be best to start diverging sooner than later.

[zamazan4ik]

@jszwedko as @ryn9 mentioned earlier, it is not possible since Elasticsearch team explicitly broken OpenSearch support in their libraries.

[jszwedko]

Forgive me for not knowing what's under the hood - but please know that elastic supplied libraries after 7.14 have be modified to specifically not work with OpenSearch. So if you are using these libraries - it would be best to start diverging sooner than later.

Ah, yes, meant to mention in my other comment that we don't rely on any SDKs for Elasticsearch but just make HTTP calls directly using the hyper crate.

[zamazan4ik]

Hmmm, in this case I tend to agree with @jszwedko approach to create opensearch sink which just wraps with some defaults an existing elasticsearch sink. Later, if will be a need, we will be able to divide opensearch and elasticsearch sinks step by step.

[ryn9]

Eventually there will be feature divergence for ingestion. When that the time comes - the OpenSearch project does maintain this library https://github.com/opensearch-project/opensearch-rs. The OpenSearch maintainers would love to hear your feedback on it and/or have you speak at one of their meetups: https://www.meetup.com/opensearch/

EDIT:
And that lib has aws sigv4 built in :)

[zamazan4ik]

@ryn9 you are right - eventually these projects will diverge a lot. We right now could start with already implemented elasticsearch sink and just wrap it as opensearch. Later, step by step, we can rewrite it with Opensearch-specific details in mind. E.g. start to use opensearch-rs in opensearch sink instead of raw hyper-based requests.

[zamazan4ik]

@ryn9 by the way, did you already try to use elasticsearch sink with Opensearch installation? Did you notice any problem?

[ryn9]

Apologies - I have not tried for a while - but I believe it was working when I last tested against an OpenSearch 1.x release

[protochron]

I use the existing elasticsearch sink with OpenSearch 1.x in AWS and it works fine, with the caveat that I handle signing separately

[ryn9]

@jszwedko when the elasticsearch output code is updated is it also being tested against opensearch?

I see that in 0.26 the following change was made to vector:
-The elasticsearch sink now supports an api_version option to specify the API version the targeted Elasticsearch instance exposes. This replaces and deprecates the suppress_type_name option which was previously used for controlling Elasticsearch version compatibility.
-It can be set to auto to attempt to automatically determine the Elasticsearch version by querying the Elasticsearch version endpoint.

Opensearch 2.x mimics the Elasticsearch 7.x line protocol - but like Elasticsearch 8.x - will not accept _type.
I am not sure what other changes are applied when using Elasticsearch 8.x line protocol (ie - if it is just _type removal), but at this point, without testing, I cannot be sure 0.26 is compatible with Opensearch 2.x

[jszwedko]

@jszwedko when the elasticsearch output code is updated is it also being tested against opensearch?

I see that in 0.26 the following change was made to vector: -The elasticsearch sink now supports an api_version option to specify the API version the targeted Elasticsearch instance exposes. This replaces and deprecates the suppress_type_name option which was previously used for controlling Elasticsearch version compatibility. -It can be set to auto to attempt to automatically determine the Elasticsearch version by querying the Elasticsearch version endpoint.

Opensearch 2.x mimics the Elasticsearch 7.x line protocol - but like Elasticsearch 8.x - will not accept _type. I am not sure what other changes are applied when using Elasticsearch 8.x line protocol (ie - if it is just _type removal), but at this point, without testing, I cannot be sure 0.26 is compatible with Opensearch 2.x

Setting api_version to 7 should suppress _type as well. Only api_version 6 should send it.

I see OpenSearch has a docker image, https://hub.docker.com/r/opensearchproject/opensearch, so it seemingly wouldn't be too hard to add it to our integration tests to ensure continued compatibility.

[ryn9]

For anyone stumbling upon this thread ... writing back to confirm that 0.26 is suppressing _type to opensearch 2.3, and successfully pushing messages.

live with a config that looks like this..

  sink_opensearch:
    type: elasticsearch
    inputs:
      - transform_remap_for_opensearch
    compression: gzip
    healthcheck: false
    endpoints:
      - "https://<CLUSTERNAME>.<REGION>.es.amazonaws.com:443"
    auth:
      strategy: "basic"
      user: "<USERNAME>"
      password: "<PASSWORD>"
    distribution:
      retry_max_duration_secs: 300
    bulk:
      index: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V"

[nike21oct]

Hi, I have a AWS opensearch cluster on AWS which is having fine grained access control enabled which is having credential and EKS cluster which is having vector installed so implemented the below syntax in configmap of vector to get the logs on opensearch. Is my syntax is correct for this implementation as I cannot able to see any index or logs on opensearch.
Need your support on this.
sinks:
elasticsearch:
type: elasticsearch
inputs: [kubernetes_logs]
healthcheck: false
endpoints:
auth:
strategy: "basic"
user: ""
password: ""

[nike21oct]

is it possible to pass the username and password in configmap as an secret , because it is not good idea to keep the credential directly into configmap

[jszwedko]

is it possible to pass the username and password in configmap as an secret , because it is not good idea to keep the credential directly into configmap

I think you can use normal Kubernetes secrets mechanisms unless I'm missing something.

[nike21oct]

I have another question
i have a vector installed in EKS cluster which is sending logs to AWS opensearch but when i see index in opensearch it is showing index only with name vector, so is there any way to configure index in config file of vector and the same we can see in opensearch.
Can you please help me into this?

[ryn9]

I have another question i have a vector installed in EKS cluster which is sending logs to AWS opensearch but when i see index in opensearch it is showing index only with name vector, so is there any way to configure index in config file of vector and the same we can see in opensearch. Can you please help me into this?

https://vector.dev/docs/reference/configuration/sinks/elasticsearch/#bulk.index

[nike21oct]

Hello , i have configured vector in kubernetes cluster and it is taking kubernetes logs as a source and sinks as a elasticsearch, so just wanted to know is logs transferring to elasticsearch instantly ?
As i cannot see latest logs of my applications it is showing old logs for one month old
can you please help me into this?

[bruceg]

Technically, Vector won't be sending it instantly, but it should be close enough given the above configuration. The default batch timeout for elasticsearch is just one second, after which it would send anything that has been queued up.

If you run vector with debugging enabled, do you see requests being sent out to the elasticsearch server?

[nike21oct]

For anyone stumbling upon this thread ... writing back to confirm that 0.26 is suppressing _type to opensearch 2.3, and successfully pushing messages.

live with a config that looks like this..

  sink_opensearch:
    type: elasticsearch
    inputs:
      - transform_remap_for_opensearch
    compression: gzip
    healthcheck: false
    endpoints:
      - "https://<CLUSTERNAME>.<REGION>.es.amazonaws.com:443"
    auth:
      strategy: "basic"
      user: "<USERNAME>"
      password: "<PASSWORD>"
    distribution:
      retry_max_duration_secs: 300
    bulk:
      index: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V"

can we create multiple index from this like:
bulk:
index: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V"
index2: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V"

Is it possible to have multiple index?

[ryn9]

can we create multiple index from this like: bulk: index: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V" index2: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V"

Is it possible to have multiple index?

You need to create multiple sinks, each with their own index definition

[spencergilbert]

can we create multiple index from this like: bulk: index: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V" index2: "{{ field1 }}-{{ field2 }}-{{ field3 }}-%G.%V"

Is it possible to have multiple index?

As @ryn9 pointed out, that's not a valid configuration. However I don't understand what you're trying to do with the two indices that the template fields don't already do. Any unique set of field1, field2, field3 (and the timestamp) will create a new index.

Additionally, please open a new Discussion for questions unrelated to the original issue - thanks.

[spencergilbert]

Adding this issue as a difference between OS and ES we need to handle:
#17690

[sandervandegeijn]

I'm using the elastic sink with the v7 api definition. It does work, but we would welcome an specific opensearch sink as well :)