feat: Custom Flattening for OTEL logs, metrics and traces #1043

nikhilsinhaparseable · 2024-12-19T11:36:22Z

add proto files for metrics and trace
add compiled rust files for metrics and trace protobuf files
add separate handlers for OTEL logs, metrics and traces
custom flattening added for OTEL logs, metrics and traces

use endpoints
/v1/logs for OTEL logs
/v1/metrics for OTEL metrics
/v1/traces for OTEL traces

add custom header X-P-Log-Source when using endpint api/v1/ingest
otel-logs for OTEL logs
otel-metrics for OTEL metrics
otel-traces for OTEL traces

de-sh · 2024-12-25T14:15:32Z

@nikhilsinhaparseable can you document the code with expectations similar to this: https://github.com/parseablehq/parseable/pull/1024/files#diff-c45a9621970d114a8f2791b564a0a3dd8bf7825dcf549fae486a98032723c12eR41-R57

The PR is extremely hard to review, given the exploding scope of code, including generated code, I would heavily suggest we consider moving this to the build stage and require build machines to have protoc and hence not have to include the code in the repo and depend on the tonic code generator to do the job of maintaining the code for us.

Please also move files into related module e.g. otel_logs.rs ~> otel/logs.rs

As for fixing the clippy, this issue is caused by including generated code into the repo, which is a bad practice, but given our circumstance, as a short term fix, we can use this:

#[allow(clippy::all)]
pub mod proto;

Cargo.toml

de-sh · 2024-12-26T05:49:00Z

We can consider using https://crates.io/crates/opentelemetry-proto

coveralls · 2024-12-26T18:43:56Z

Pull Request Test Coverage Report for Build 12621906304

Details

89 of 1066 (8.35%) changed or added relevant lines in 11 files are covered.
4 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.3%) to 12.483%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/kafka.rs	0	1	0.0%
src/handlers/http/modal/server.rs	0	3	0.0%
src/utils/json/mod.rs	11	14	78.57%
src/event/format/mod.rs	2	10	20.0%
src/handlers/http/modal/utils/ingest_utils.rs	8	32	25.0%
src/handlers/http/ingest.rs	58	129	44.96%
src/otel/logs.rs	0	110	0.0%
src/otel/otel_utils.rs	0	117	0.0%
src/otel/traces.rs	0	220	0.0%
src/otel/metrics.rs	0	420	0.0%

Files with Coverage Reduction	New Missed Lines	%
src/kafka.rs	4	0.0%

Totals
Change from base Build 12621383951:	-0.3%
Covered Lines:	2467
Relevant Lines:	19763

💛 - Coveralls

nikhilsinhaparseable · 2024-12-27T03:28:58Z

@de-sh removed the pre-compiled and the protobuf files.
I am using the crate you suggested.
Please review again.

Cargo.toml

nitisht · 2024-12-29T04:08:31Z

Is this ready now for merge?

de-sh · 2025-01-04T10:31:49Z

src/event/format/json.rs

@@ -52,8 +52,17 @@ impl EventFormat for Event {
        static_schema_flag: Option<&String>,
        time_partition: Option<&String>,
        schema_version: SchemaVersion,
+        log_source: &str,


We should move away from using str given how wide the possibilities are, please consider using enums similar to what we are doing with SchemaVersion above, it will also make things easier to read

de-sh · 2025-01-05T08:21:58Z

src/otel/otel_utils.rs

+            value_json.insert(key.to_string(), Value::Bool(bool_val));
+        }
+        OtelValue::IntValue(int_val) => {
+            value_json.insert(key.to_string(), Value::String(int_val.to_string()));


Can you remind me why we are doing int to string, instead of number?

otel collector sends int value as string, but now it can be serialised as int, hence, i think we can now change it to number, let me test it out

nitisht · 2025-01-05T17:03:58Z

@nikhilsinhaparseable can you resolve the conflicts

add proto files for metrics and trace add compiled rust files for metrics and trace protobuf files add separate handlers for OTEL logs, metrics and traces custom flattening added for OTEL logs and metrics

add custom flattening for otel traces use endpoints `/v1/logs` for OTEL logs `/v1/metrics` for OTEL metrics `/v1/traces` for OTEL traces add custom header X-P-Log-Source when using endpint `api/v1/ingest` `otel-logs` for OTEL logs `otel-metrics` for OTEL metrics `otel-traces` for OTEL traces

Co-authored-by: Devdutt Shenoi <[email protected]> Signed-off-by: Nikhil Sinha <[email protected]>

used `opentelemetry-proto` crate deleted compiled rust files and protobuf files

Co-authored-by: Devdutt Shenoi <[email protected]> Signed-off-by: Nikhil Sinha <[email protected]>

… metrics

…q#1043) custom flattening for OTEL data add proto files for metrics and trace add compiled rust files for metrics and trace protobuf files add separate handlers for OTEL logs, metrics and traces custom flattening added for OTEL logs and metrics custom flattening for OTEL traces use endpoints `/v1/logs` for OTEL logs `/v1/metrics` for OTEL metrics `/v1/traces` for OTEL traces add custom header X-P-Log-Source when using endpint `api/v1/ingest` `otel-logs` for OTEL logs `otel-metrics` for OTEL metrics `otel-traces` for OTEL traces --------- Signed-off-by: Nikhil Sinha <[email protected]> Co-authored-by: Devdutt Shenoi <[email protected]>

nitisht requested a review from de-sh December 19, 2024 11:37

nikhilsinhaparseable force-pushed the otel-metrics-traces-flattening branch from a80beb2 to 6b4b20d Compare December 25, 2024 09:39

nikhilsinhaparseable force-pushed the otel-metrics-traces-flattening branch from 6b4b20d to 570a7f4 Compare December 25, 2024 19:09

de-sh reviewed Dec 26, 2024

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

nitisht added the for next release label Dec 27, 2024

nikhilsinhaparseable force-pushed the otel-metrics-traces-flattening branch from 75370b6 to 95a915f Compare December 27, 2024 05:09

de-sh reviewed Dec 27, 2024

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

nikhilsinhaparseable force-pushed the otel-metrics-traces-flattening branch 3 times, most recently from 1ef9619 to 74ddc86 Compare January 4, 2025 02:47

de-sh reviewed Jan 4, 2025

View reviewed changes

de-sh reviewed Jan 5, 2025

View reviewed changes

de-sh approved these changes Jan 5, 2025

View reviewed changes

nikhilsinhaparseable and others added 11 commits January 5, 2025 12:39

feat: custom flattening for OTEL data

53ad2ba

add proto files for metrics and trace add compiled rust files for metrics and trace protobuf files add separate handlers for OTEL logs, metrics and traces custom flattening added for OTEL logs and metrics

update data type based on data from otel collector

90b2433

fix: refactor, deepsource fixes

30625b0

fix: cargo hack, incorporate review comments

95c2c1e

Update Cargo.toml

04347bf

Co-authored-by: Devdutt Shenoi <[email protected]> Signed-off-by: Nikhil Sinha <[email protected]>

fix: removed pre-compiled otel files

4a45a5d

used `opentelemetry-proto` crate deleted compiled rust files and protobuf files

restructure: moved otel module from handlers to src

a74bb0b

suggestions

6eea205

Update Cargo.toml

4bcf8f0

Co-authored-by: Devdutt Shenoi <[email protected]> Signed-off-by: Nikhil Sinha <[email protected]>

dependency: update opentelemetry-proto dependency to parseable fork

e6ad4f0

nikhilsinhaparseable and others added 11 commits January 5, 2025 12:44

fix: trace_id, span_id mapping

b8a9416

transform: epoch nanosecond number to timestamp

e83ed88

bucket counts, explicit bounds, quantile values as json array in OTEL…

1681a75

… metrics

restrict /api/v1/ingest api for otel

fa70652

restrict generic flattening for otel data

47bbb80

fix: create metric record in case of empty data

67b4c5a

fix: corrected attribute-intValue to json number

189197c

refactor: LogSource enum

2f091cb

fix: generic flattening only for non otel sources

ef0f4e7

linter fix

ae8f826

fix: rebase from main

6908630

nikhilsinhaparseable force-pushed the otel-metrics-traces-flattening branch from 59fac1d to 6908630 Compare January 5, 2025 18:20

nitisht merged commit 622b9a2 into parseablehq:main Jan 6, 2025
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Custom Flattening for OTEL logs, metrics and traces #1043

feat: Custom Flattening for OTEL logs, metrics and traces #1043

nikhilsinhaparseable commented Dec 19, 2024

de-sh commented Dec 25, 2024 •

edited

Loading

de-sh commented Dec 26, 2024

coveralls commented Dec 26, 2024 •

edited

Loading

nikhilsinhaparseable commented Dec 27, 2024

nitisht commented Dec 29, 2024

de-sh Jan 4, 2025

de-sh Jan 5, 2025

nikhilsinhaparseable Jan 5, 2025

nitisht commented Jan 5, 2025

feat: Custom Flattening for OTEL logs, metrics and traces #1043

feat: Custom Flattening for OTEL logs, metrics and traces #1043

Conversation

nikhilsinhaparseable commented Dec 19, 2024

de-sh commented Dec 25, 2024 • edited Loading

de-sh commented Dec 26, 2024

coveralls commented Dec 26, 2024 • edited Loading

Pull Request Test Coverage Report for Build 12621906304

Details

💛 - Coveralls

nikhilsinhaparseable commented Dec 27, 2024

nitisht commented Dec 29, 2024

de-sh Jan 4, 2025

Choose a reason for hiding this comment

de-sh Jan 5, 2025

Choose a reason for hiding this comment

nikhilsinhaparseable Jan 5, 2025

Choose a reason for hiding this comment

nitisht commented Jan 5, 2025

de-sh commented Dec 25, 2024 •

edited

Loading

coveralls commented Dec 26, 2024 •

edited

Loading