Supports OpenSearch V2 through ES_* environment variables

[reta]

Add Zipkin support of OpenSearch storage

The Elasticsearch storage now supports OpenSearch as a backed as well. The implementation relies on distribution property that is returned as part of Elasticsearch / OpenSearch HTTP GET / API. Please note that the storage version is now abstracted as BaseVersion with two implementation: ElasticsearchVersion and OpensearchVersion.

Although OpenSearch is a fork of Elasticsearch as of 7.10.2, the projects diverged sufficiently far from each other. Luckily, Zipkin relies on the features that have not been impacted (so far) and work the same way across both projects.

• Add javadocs
• Add documentation

[codefromthecrypt]

I think it is easiest to make the image first in a separate PR. Then, merge and we can use it.

I don't recall the history of the feature ask, but if someone hasn't asked for 1.x maybe better to only do 2.x because there's a linear time increase per images. Usually we don't add old versions as a first storage option.

At any rate, you would do a PR for the image, and make sure it is in the docker readme test workflows etc. When that merges, after master builds clean, you can re-cut the last version docker images as docker-x.x.x. Finally, as it is a new image, it will be private by default, so you need to click to make it public https://github.com/orgs/openzipkin/packages

[codefromthecrypt]

@reta for the error around ghcr.io/openzipkin/zipkin-opensearch2:3.3.1 you want to push a tag docker-3.3.1 off latest master. this will re-publish and also include the new one, unless we missed a spot somewhere.

[codefromthecrypt]

also don't forget to set this public https://github.com/orgs/openzipkin/packages/container/zipkin-opensearch2/settings

[reta]

@reta for the error around ghcr.io/openzipkin/zipkin-opensearch2:3.3.1 you want to push a tag docker-3.3.1 off latest master. this will re-publish and also include the new one, unless we missed a spot somewhere.

Thanks @codefromthecrypt , it seems like we need to make a 3.3.1 release first since Docker images are looking for Zipkin 3.3.1 bits ...

[codefromthecrypt]

@reta oh sorry.. I didn't remember what release was current. change the test code to use 3.3.0 and re-cut docker-3.3.0. that's the simplest way and will get the new test image

[reta]

also don't forget to set this public https://github.com/orgs/openzipkin/packages/container/zipkin-opensearch2/settings

@codefromthecrypt apologies, I see the image is 100% pushed but I cannot see it in the Packages, is it my permissions? (because it seems like I cannot change visibility level for existing packages as well) I see that I could change the visibility for existing packages

[codefromthecrypt]

switched visibility to public ;) https://github.com/orgs/openzipkin/packages/container/zipkin-opensearch2/settings

[codefromthecrypt]

when ready, re-prep the PR description, after making sure any markdown that need a usage example update have been so. Also, anything in general READMEs about things we test mention opensearch since after this, it is true.

[codefromthecrypt]

also bump poms 3.3.1-SNAPSHOT -> 3.4.0-SNAPSHOT as def not a patch ;)

[codefromthecrypt]

also I think a question originally was about hosted aws opensearch (zipkin-aws). want to try integrating with that via snapshot locally prior to mark this for review?

[reta]

also I think a question originally was about hosted aws opensearch (zipkin-aws). want to try integrating with that via snapshot locally prior to mark this for review?

Absolutely, thanks @codefromthecrypt !

[reta]

also I think a question originally was about hosted aws opensearch (zipkin-aws). want to try integrating with that via snapshot locally prior to mark this for review?

@codefromthecrypt I have to admit failure to verify the flow, I was not able to get AWS managed OpenSearch instance(s) for testing purposes. On an optimistic note, AFAIK the managed OpenSearch uses the same open source distribution and should work seamlessly.

[codefromthecrypt]

okie dokie. I'm a bit swamped, but if I've not reviewed this by saturday, nag me as I should be able to get time to look closely by then.

[codefromthecrypt]

Thanks for doing this! As far as I understand, this makes a heuristic internally to detect OS vs ES, so that the server config pretty much remains the same. Do you know if zipkin-dependencies works with OS using the current driver? Because merging this without dependencies working will immediately lead to another problem.

p.s. maybe cc anyone who has asked for OS on this PR so that they can know about it directly

[codefromthecrypt]

thanks for the detailed updates here

[codefromthecrypt]

I forget.. this appears duplicated and is using the method elasticsearch() not the field. Can this be pulled up?

[reta]

Sorry, you mean replace elasticsearch() -> elasticsearch? (or better to say opensearch in this case)

[reta]

Do you know if zipkin-dependencies works with OS using the current driver?

Ah, thanks a lot for pointing it out @codefromthecrypt, it won't work with Elasticsearch, we will need to add OpenSearch module alongside the Elasticsearch one.

[reta]

Hm ... lint check failure is new:

 build-bin/configure_lint: 5: markdown-link-check: not found

added 66 packages in 3s

16 packages are looking for funding
  run `npm fund` for details
usage: yamllint [-h] [-] [-c CONFIG_FILE | -d CONFIG_DATA] [--list-files]
                [-f {parsable,standard,colored,github,auto}] [-s]
                [--no-warnings] [-v]
                [FILE_OR_DIR ...]
yamllint: error: one of the arguments FILE_OR_DIR - is required

[codefromthecrypt]

let's hold merging this until that's sorted out as I don't want to have a new storage option that doesn't work due to zipkin-dependencies (bomb)

[reta]

let's hold merging this until that's sorted out as I don't want to have a new storage option that doesn't work due to zipkin-dependencies (bomb)

I could pick it up later this week, very likely over the weekend (the good thing, AFAIK OpenSearch has Spark / Hadoop support on par with Elasticsearch)

[codefromthecrypt]

thanks @reta!

[reta]

And #3634 strikes again

[reta]

@codefromthecrypt this is super funny:

 ERROR: 1 dead links found!
  [✖] https://docs.datastax.com/en/developer/java-driver/4.17/manual/core/request_tracker/#request-logger → Status: 404

The link opens ... but:

$ curl -iv https://docs.datastax.com/en/developer/java-driver/4.17/manual/core/request_tracker/#request-logger

HTTP/1.1 404 Not Found
Content-Type: text/html
Content-Length: 36054
Connection: keep-alive
Last-Modified: Wed, 22 May 2024 18:02:05 GMT
x-amz-server-side-encryption: AES256
Accept-Ranges: bytes
Server: AmazonS3
Date: Fri, 24 May 2024 18:08:32 GMT
ETag: "1e69f79408df5da533fcc320698669a9"
Vary: Accept-Encoding
X-Cache: Error from cloudfront
Via: 1.1 10f978be0e76903f65b35bee6581c310.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: YUL62-P2
X-Amz-Cf-Id: w5XQ_h8cv8834R5pBln1oIEpVltBAX6SnR56UMhk6f3F-cjtj0diVw==
Age: 73337
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Content-Security-Policy: child-src 'self'; frame-src youtube.com https://www.youtube.com; worker-src 'self' blob:
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000

<!DOCTYPE html>
<html lang="en">
  <head>
 ....

[codefromthecrypt]

this is super funny:

yes, no PR is complete without unrelated bruising from lint checkers ;)

[codefromthecrypt]

ok! finally time to ship this 👍