feat(openobserve transform): Add OpenObserve as an officially supported sink #21531

chaitanya-sistla · 2024-10-17T04:17:16Z

This PR adds OpenObserve as an officially supported sink in Vector's documentation. OpenObserve utilizes an HTTP-based API for data ingestion, and this PR proposes updating the documentation with an example configuration for users to follow.

No new functionality is required, as the current HTTP sink already serves the purpose effectively. The primary goal is to improve user experience by offering clear guidance for setting up Vector with OpenObserve, ensuring confidence in its usage as a supported observability solution.

type = "http"
inputs = [ "source_or_transform_id" ]
uri = "http://localhost:5080/api/default/default/_json"
method = "post"
auth.strategy = "basic"
auth.user = "[email protected]"
auth.password = "v9Ca7qHaMKELlDtU" (fake)
compression = "gzip"
encoding.codec = "json"
encoding.timestamp_format = "rfc3339"
healthcheck.enabled = false

Why This Matters?
By officially documenting OpenObserve as a supported sink, we provide clear setup instructions for users, allowing them to confidently configure Vector for OpenObserve.

Closes: #21514

bits-bot · 2024-10-17T04:17:19Z

All committers have signed the CLA.

chaitanya-sistla · 2024-10-17T14:36:10Z

@jszwedko could you help me review this? Is anything pending from my end?

pront

Hello, this PR doesn't require any code changes and it seems more like a user guide on how to configure an existing sink. It feels like https://github.com/vectordotdev/vector-demos might be the right place for this. Or it could live under a new Community section in https://vector.dev/guides/

Cargo.toml

* implement SinkConfig for OpenObserveConfig * chore: add comments and update reference samples * chore: add openobserve to urls and services

pront · 2024-10-23T18:26:52Z

src/sinks/openobserve.rs

+    async fn build(&self, cx: SinkContext) -> crate::Result<(VectorSink, Healthcheck)> {
+        let request = self.request.clone();
+
+        // OpenObserve supports native HTTP ingest endpoint. This configuration wraps


I would move this above SinkConfig and rephrase as:

/// This sink wraps the Vector HTTP sink to provide official support for OpenObserve's /// native HTTP ingest endpoint. By doing so, it maintains a distinct configuration for /// the OpenObserve sink, separate from the Vector HTTP sink. This approach ensures /// that future changes to OpenObserve's interface can be accommodated without impacting /// the underlying Vector HTTP sink.

pront · 2024-10-23T18:28:10Z

Makefile

@@ -367,7 +367,7 @@ test-integration: ## Runs all integration tests
 test-integration: test-integration-amqp test-integration-appsignal test-integration-aws test-integration-axiom test-integration-azure test-integration-chronicle test-integration-clickhouse
 test-integration: test-integration-databend test-integration-docker-logs test-integration-elasticsearch
 test-integration: test-integration-eventstoredb test-integration-fluent test-integration-gcp test-integration-greptimedb test-integration-humio test-integration-http-client test-integration-influxdb
-test-integration: test-integration-kafka test-integration-logstash test-integration-loki test-integration-mongodb test-integration-nats
+test-integration: test-integration-kafka test-integration-logstash test-integration-loki test-integration-mongodb test-integration-nats test-integration-openobserve


I don't see any use of test-integration-openobserve in this PR. Can we delete it?

pront · 2024-10-23T18:29:06Z

Cargo.toml

@@ -885,6 +888,7 @@ mqtt-integration-tests = ["sinks-mqtt"]
 nats-integration-tests = ["sinks-nats", "sources-nats"]
 nginx-integration-tests = ["sources-nginx_metrics"]
 opentelemetry-integration-tests = ["sources-opentelemetry", "dep:prost"]
+openobserve-integration-tests = ["sinks-openobserve"]


I don't see any use of openobserve-integration-tests in this PR. Can we delete it here and from the github workflows?

pront · 2024-10-28T19:51:44Z

.github/workflows/changes.yml

@@ -104,6 +104,8 @@ on:
        value: ${{ jobs.int_tests.outputs.nginx }}
      opentelemetry:
        value: ${{ jobs.int_tests.outputs.opentelemetry }}
+      openobserve:


int_tests is a GitHub job that detects if a branch contains changes to integration tests. Since you didn't added any integration tests dedicated to openobserve you should revert all changes to .github/workflows/*,

pront

👋 Left some nits. Since it affects the generated docs, you can run make generate-component-docs && make check-component-docs to regren and validate.

Makefile

src/sinks/openobserve.rs

Apply suggestions Co-authored-by: Pavlos Rontidis <[email protected]>

pront

Thank you, this looks almost ready to merge 🚀

One comment here.

removing the trailing space

chaitanya-sistla · 2024-10-31T19:51:39Z

I have removed the space @pront

pront · 2024-11-06T19:24:36Z

Please add a changelog entry. You can find details on this document.

chaitanya-sistla · 2024-11-07T02:48:18Z

added @pront

changelog.d/21531_add_openobserve_as_new_sink.md

Co-authored-by: Pavlos Rontidis <[email protected]>

pront · 2024-11-08T17:21:08Z

Let's add a new "openobserve transform" scope somewhere here: https://github.com/vectordotdev/vector/blob/master/.github/workflows/semantic.yml#L182

pront · 2024-11-08T17:30:47Z

Also, please see

vector/docs/DOCUMENTING.md

Line 89 in ac1e975

CI=true make check-docs

chaitanya-sistla requested review from a team as code owners October 17, 2024 04:17

github-actions bot added the domain: external docs Anything related to Vector's external, public documentation label Oct 17, 2024

pront reviewed Oct 17, 2024

View reviewed changes

Cargo.toml Show resolved Hide resolved

neko-dd approved these changes Oct 17, 2024

View reviewed changes

chaitanya-sistla force-pushed the feature/openobserve-sink branch 9 times, most recently from d9abffa to e9bd1fe Compare October 18, 2024 14:14

github-actions bot added the domain: sinks Anything related to the Vector's sinks label Oct 20, 2024

chaitanya-sistla marked this pull request as draft October 20, 2024 18:42

chaitanya-sistla marked this pull request as ready for review October 21, 2024 01:30

chaitanya-sistla marked this pull request as draft October 21, 2024 01:30

chaitanya-sistla and others added 8 commits October 20, 2024 20:45

adding openobserve as a sink connector

2a46d42

adding placeholder instead of fake password

6857318

adding openobserve to allow.txt

734a2e9

adding openobserve to allow.txt

ad1cc57

updated CI and minor progress

4d8db24

implement SinkConfig for OpenObserveConfig (#1)

835cdee

* implement SinkConfig for OpenObserveConfig * chore: add comments and update reference samples * chore: add openobserve to urls and services

rename uri

378e0db

update: allow default uri as openobserve ingestion endpoint

9308824

pront reviewed Oct 23, 2024

View reviewed changes

accommodated feedback

552c0ef

pront reviewed Oct 28, 2024

View reviewed changes

removed changes related to integration tests

1a52f71

pront approved these changes Oct 30, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

src/sinks/openobserve.rs Outdated Show resolved Hide resolved

src/sinks/openobserve.rs Show resolved Hide resolved

src/sinks/openobserve.rs Show resolved Hide resolved

prabhatsharma and others added 2 commits October 30, 2024 10:43

Apply suggestions from code review

e74b922

Apply suggestions Co-authored-by: Pavlos Rontidis <[email protected]>

remove unused line

3d1a42a

pront reviewed Oct 31, 2024

View reviewed changes

Update Makefile

93c0dd4

removing the trailing space

pront approved these changes Oct 31, 2024

View reviewed changes

pront enabled auto-merge October 31, 2024 20:09

chaitanya-sistla changed the title ~~Add OpenObserve as an officially supported sink~~ feat: Add OpenObserve as an officially supported sink Nov 5, 2024

removing trailing spaces

cac339e

auto-merge was automatically disabled November 5, 2024 02:42
Head branch was pushed to by a user without write access

pront enabled auto-merge November 5, 2024 15:59

Create 21531_add_openobserve_as_new_sink.md

56a047e

auto-merge was automatically disabled November 7, 2024 02:47
Head branch was pushed to by a user without write access

pront reviewed Nov 7, 2024

View reviewed changes

changelog.d/21531_add_openobserve_as_new_sink.md Outdated Show resolved Hide resolved

Update changelog.d/21531_add_openobserve_as_new_sink.md

2b18728

Co-authored-by: Pavlos Rontidis <[email protected]>

pront enabled auto-merge November 7, 2024 19:25

changing the file name

11bb415

auto-merge was automatically disabled November 8, 2024 03:11
Head branch was pushed to by a user without write access

pront enabled auto-merge November 8, 2024 16:18

pront disabled auto-merge November 8, 2024 16:18

pront changed the title ~~feat: Add OpenObserve as an officially supported sink~~ feat(openobserve transform): Add OpenObserve as an officially supported sink Nov 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(openobserve transform): Add OpenObserve as an officially supported sink #21531

feat(openobserve transform): Add OpenObserve as an officially supported sink #21531

chaitanya-sistla commented Oct 17, 2024 •

edited by jszwedko

Loading

bits-bot commented Oct 17, 2024 •

edited

Loading

chaitanya-sistla commented Oct 17, 2024

pront left a comment

pront Oct 23, 2024

prabhatsharma Oct 26, 2024

pront Oct 23, 2024

prabhatsharma Oct 26, 2024

pront Oct 23, 2024

prabhatsharma Oct 26, 2024

pront Oct 28, 2024

prabhatsharma Oct 29, 2024

pront left a comment

pront left a comment

chaitanya-sistla commented Oct 31, 2024

pront commented Nov 6, 2024

chaitanya-sistla commented Nov 7, 2024

pront commented Nov 8, 2024

pront commented Nov 8, 2024

feat(openobserve transform): Add OpenObserve as an officially supported sink #21531

Are you sure you want to change the base?

feat(openobserve transform): Add OpenObserve as an officially supported sink #21531

Conversation

chaitanya-sistla commented Oct 17, 2024 • edited by jszwedko Loading

bits-bot commented Oct 17, 2024 • edited Loading

chaitanya-sistla commented Oct 17, 2024

pront left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pront left a comment

Choose a reason for hiding this comment

pront left a comment

Choose a reason for hiding this comment

chaitanya-sistla commented Oct 31, 2024

pront commented Nov 6, 2024

chaitanya-sistla commented Nov 7, 2024

pront commented Nov 8, 2024

pront commented Nov 8, 2024

chaitanya-sistla commented Oct 17, 2024 •

edited by jszwedko

Loading

bits-bot commented Oct 17, 2024 •

edited

Loading