Datadog exporter improvements #5609

BrynCooke · 2024-07-04T14:22:17Z

This PR updates the Datadog exporter with a backport from the latest code.
The exporter code has been placed in the apollo-router/src/plugins/telemetry/tracing/datadog_exporter directory.

It also fixes:

span.kind not being propagated
Missing span stats. (Users may override configuration for these).

Span metrics are enabled for major built in spans by default.

telemetry:
  exporters:
    tracing:
      experimental_response_trace_id:
        enabled: true
        header_name: apollo-custom-trace-id
        format: datadog
      common:
        service_name: router
      datadog:
        enabled: true
        batch_processor:
          scheduled_delay: 100ms
        span_metrics: # New config option to enable or disable span metrics. This is a free format span to boolean map.
          supergraph: false

  instrumentation:
    spans:
      mode: spec_compliant
      supergraph:
        attributes:
          graphql.operation.name: true

Checklist

Complete the checklist (and note appropriate exceptions) before the PR is marked ready-for-review.

Exceptions

Note any exceptions here

Notes

It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. ↩
Configuration is an important part of many changes. Where applicable please try to document configuration examples. ↩
Tick whichever testing boxes are applicable. If you are adding Manual Tests, please document the manual testing (extensively) in the Exceptions. ↩

router-perf · 2024-07-04T14:22:50Z

…e page.

garypen

Configuration comment: When we update otel to the latest library, what will be the impact on this configuration that we're adding. Will it continue to work unchanged or will it become redundant or ...?

garypen · 2024-07-08T11:11:13Z

.changesets/fix_bryn_datadog_exporter_span_kind.md

+### Datadog `span.kind` now populated ([PR #5609](https://github.com/apollographql/router/pull/5609))
+
+Datadog traces use `span.kind` to differentiate between different types of spans. 
+This change ensures that the `span.kind` is correctly populated using the Open Telemetry span kind which has a 1-2-1 mapping to thouse set out in [dd-trace](https://github.com/DataDog/dd-trace-go/blob/main/ddtrace/ext/span_kind.go).


Suggested change

This change ensures that the `span.kind` is correctly populated using the Open Telemetry span kind which has a 1-2-1 mapping to thouse set out in [dd-trace](https://github.com/DataDog/dd-trace-go/blob/main/ddtrace/ext/span_kind.go).

This change ensures that the `span.kind` is correctly populated using the Open Telemetry span kind which has a 1-2-1 mapping to those set out in [dd-trace](https://github.com/DataDog/dd-trace-go/blob/main/ddtrace/ext/span_kind.go).

garypen · 2024-07-08T11:11:26Z

.changesets/fix_bryn_datadog_exporter_span_metrics.md

@@ -0,0 +1,32 @@
+### Datadog span metrics now supported ([PR #5609](https://github.com/apollographql/router/pull/5609))


Suggested change

### Datadog span metrics now supported ([PR #5609](https://github.com/apollographql/router/pull/5609))

### Datadog span metrics are now supported ([PR #5609](https://github.com/apollographql/router/pull/5609))

garypen · 2024-07-08T11:12:08Z

.changesets/fix_bryn_datadog_exporter_span_metrics.md

@@ -0,0 +1,32 @@
+### Datadog span metrics now supported ([PR #5609](https://github.com/apollographql/router/pull/5609))
+
+When using the APM view in Datadog, span metrics will be displayed for any span that was a top level span or has been explicitly has `_dd.measured` flag set.


Suggested change

When using the APM view in Datadog, span metrics will be displayed for any span that was a top level span or has been explicitly has `_dd.measured` flag set.

When using the APM view in Datadog, span metrics will be displayed for any span that was a top level span or has the `_dd.measured` flag set.

garypen · 2024-07-08T11:15:54Z

apollo-router/src/plugins/telemetry/consts.rs

@@ -18,8 +18,9 @@ pub(crate) const REQUEST_SPAN_NAME: &str = "request";
 pub(crate) const QUERY_PLANNING_SPAN_NAME: &str = "query_planning";
 pub(crate) const HTTP_REQUEST_SPAN_NAME: &str = "http_request";
 pub(crate) const SUBGRAPH_REQUEST_SPAN_NAME: &str = "subgraph_request";
+pub(crate) const QUERY_PARSING_SPAN_NAME: &str = "parse_query";


This looks like a "drive-by" fix. Should this be called out in the changelog as additional functionality or split into its own PR?

garypen · 2024-07-08T11:18:22Z

apollo-router/src/plugins/telemetry/tracing/datadog.rs

+    /// Which spans will be eligable for span stats to be collected for viewing in the APM view.
+    /// Defaults to true for `router` and `supergraph`, `subgraph`, `subgraph_request`, `http_request` and `query_planning`.


This comment isn't accurate. It is missing at least execution. I would expect the defaults to be as listed in BUILT_IN_SPAN_NAMES, so probably better to just reference that.

garypen · 2024-07-08T11:19:05Z

apollo-router/src/plugins/telemetry/tracing/datadog.rs

@@ -170,13 +196,97 @@ impl TracingConfigurator for Config {
                    .expect("cargo version is set as a resource default;qed")
                    .to_string(),
            )
+            // This prevents spurious errors in the log when talking to the agent


Another "drive-by", which probably merits some mention in the changelog.

garypen · 2024-07-08T11:21:06Z

docs/source/configuration/telemetry/exporters/tracing/datadog.mdx

@@ -166,7 +166,7 @@ The default resource mappings are:
 | `subgraph_request`      | `graphql.operation.name`    |
 | `http_request`          | `http.route`                |

-You may override these mappings by specifying the `resource_mapping` configuration:
+You may override these mappings by specifying `resource_mapping` configuration:


Suggested change

You may override these mappings by specifying `resource_mapping` configuration:

You may override these mappings by specifying the `resource_mapping` configuration:

bryn added 4 commits July 3, 2024 23:32

Minimal changes to support all spans being measured

2321d30

Backport latest datadog exporter

a53ab53

Lint

8b3bca2

Add exporter wrapper to support measurement

f880596

This comment has been minimized.

Sign in to view

apollo-bot2 assigned BrynCooke Jul 4, 2024

bryn and others added 5 commits July 4, 2024 16:39

Use span.kind rather than span.type

350fe61

Add tests, add missing span to consts.rs

636f571

Lint

1e7ff60

Merge branch 'dev' into bryn/datadog-exporter

e23fed8

Fix tests failing dure to missing removal of gated features

bd854c3

BrynCooke requested review from bnjjj and garypen July 5, 2024 08:27

BrynCooke marked this pull request as ready for review July 5, 2024 08:27

BrynCooke requested review from a team as code owners July 5, 2024 08:27

bryn added 2 commits July 5, 2024 09:38

Add span_metrics to docs and fix some other minor issues on the sam…

9eaa4de

…e page.

Changelog and minor docs fixes

7b8fb68

BrynCooke requested a review from a team as a code owner July 5, 2024 09:01

bryn and others added 2 commits July 5, 2024 10:09

Lint

fadfc33

Merge branch 'dev' into bryn/datadog-exporter

e573a7c

garypen reviewed Jul 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Datadog exporter improvements #5609

Datadog exporter improvements #5609

BrynCooke commented Jul 4, 2024 •

edited

Loading

This comment has been minimized.

router-perf bot commented Jul 4, 2024 •

edited by BrynCooke

Loading

garypen left a comment

garypen Jul 8, 2024

garypen Jul 8, 2024

garypen Jul 8, 2024

garypen Jul 8, 2024

garypen Jul 8, 2024

garypen Jul 8, 2024

garypen Jul 8, 2024

	This change ensures that the `span.kind` is correctly populated using the Open Telemetry span kind which has a 1-2-1 mapping to thouse set out in [dd-trace](https://github.com/DataDog/dd-trace-go/blob/main/ddtrace/ext/span_kind.go).
	This change ensures that the `span.kind` is correctly populated using the Open Telemetry span kind which has a 1-2-1 mapping to those set out in [dd-trace](https://github.com/DataDog/dd-trace-go/blob/main/ddtrace/ext/span_kind.go).

		@@ -0,0 +1,32 @@
		### Datadog span metrics now supported ([PR #5609](https://github.com/apollographql/router/pull/5609))

	### Datadog span metrics now supported ([PR #5609](https://github.com/apollographql/router/pull/5609))
	### Datadog span metrics are now supported ([PR #5609](https://github.com/apollographql/router/pull/5609))

		@@ -0,0 +1,32 @@
		### Datadog span metrics now supported ([PR #5609](https://github.com/apollographql/router/pull/5609))

		When using the APM view in Datadog, span metrics will be displayed for any span that was a top level span or has been explicitly has `_dd.measured` flag set.

		/// Which spans will be eligable for span stats to be collected for viewing in the APM view.
		/// Defaults to true for `router` and `supergraph`, `subgraph`, `subgraph_request`, `http_request` and `query_planning`.

	You may override these mappings by specifying `resource_mapping` configuration:
	You may override these mappings by specifying the `resource_mapping` configuration:

Datadog exporter improvements #5609

Are you sure you want to change the base?

Datadog exporter improvements #5609

Conversation

BrynCooke commented Jul 4, 2024 • edited Loading

Footnotes

This comment has been minimized.

router-perf bot commented Jul 4, 2024 • edited by BrynCooke Loading

garypen left a comment

Choose a reason for hiding this comment

garypen Jul 8, 2024

Choose a reason for hiding this comment

garypen Jul 8, 2024

Choose a reason for hiding this comment

garypen Jul 8, 2024

Choose a reason for hiding this comment

garypen Jul 8, 2024

Choose a reason for hiding this comment

garypen Jul 8, 2024

Choose a reason for hiding this comment

garypen Jul 8, 2024

Choose a reason for hiding this comment

garypen Jul 8, 2024

Choose a reason for hiding this comment

BrynCooke commented Jul 4, 2024 •

edited

Loading

router-perf bot commented Jul 4, 2024 •

edited by BrynCooke

Loading