Fix incorrect metrics getting generated from multiple readers #5900

pree-dew · 2024-10-19T09:52:56Z

Will fix #5866

…w, once the fix goes it will pass

…trics

…ipeline and get the measure during collect for the called pipeline

dashpole

looking great overall.

sdk/metric/meter.go

sdk/metric/pipeline.go

sdk/metric/meter.go

…trument for each pipeline. 3. remove the registercallback function

codecov · 2024-11-01T09:24:06Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.6%. Comparing base (d2b0663) to head (536497b).
Report is 1 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #5900   +/-   ##
=====================================
  Coverage   84.6%   84.6%           
=====================================
  Files        272     272           
  Lines      22857   22890   +33     
=====================================
+ Hits       19350   19380   +30     
- Misses      3163    3166    +3     
  Partials     344     344

see 5 files with indirect coverage changes

sdk/metric/meter.go

sdk/metric/pipeline.go

…test case to check concurrency on read and write of observer. 3. Remove the check of adding measure as it will get response from cache.

dashpole · 2024-11-01T16:51:05Z

Looks like this just needs a changelog, and then reviews from others!

pree-dew · 2024-11-01T17:27:51Z

@dashpole added changlog entry. I really appreciate all the help and feedback provided by you. :)

CHANGELOG.md

dashpole · 2024-11-01T17:50:21Z

Feel free to mark this ready for review once you are happy with it

…anding

dashpole

Thanks!

sdk/metric/pipeline_test.go

…trics

…mechanism

sdk/metric/pipeline_test.go

pellared · 2024-11-07T09:03:31Z

@pree-dew, can you please take a look at #5900 (comment)? Not a blocker, but we may want to have follow-up actions.

…trics

pellared · 2024-11-08T06:29:21Z

@pree-dew, thanks 🥇

pree-dew · 2024-11-08T06:38:10Z

Happy to help @pellared , waiting to learn from all of you!

### Added - Add `go.opentelemetry.io/otel/sdk/metric/exemplar.AlwaysOffFilter`, which can be used to disable exemplar recording. (#5850) - Add `go.opentelemetry.io/otel/sdk/metric.WithExemplarFilter`, which can be used to configure the exemplar filter used by the metrics SDK. (#5850) - Add `ExemplarReservoirProviderSelector` and `DefaultExemplarReservoirProviderSelector` to `go.opentelemetry.io/otel/sdk/metric`, which defines the exemplar reservoir to use based on the aggregation of the metric. (#5861) - Add `ExemplarReservoirProviderSelector` to `go.opentelemetry.io/otel/sdk/metric.Stream` to allow using views to configure the exemplar reservoir to use for a metric. (#5861) - Add `ReservoirProvider`, `HistogramReservoirProvider` and `FixedSizeReservoirProvider` to `go.opentelemetry.io/otel/sdk/metric/exemplar` to make it convenient to use providers of Reservoirs. (#5861) - The `go.opentelemetry.io/otel/semconv/v1.27.0` package. The package contains semantic conventions from the `v1.27.0` version of the OpenTelemetry Semantic Conventions. (#5894) - Add `Attributes attribute.Set` field to `Scope` in `go.opentelemetry.io/otel/sdk/instrumentation`. (#5903) - Add `Attributes attribute.Set` field to `ScopeRecords` in `go.opentelemetry.io/otel/log/logtest`. (#5927) - `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc` adds instrumentation scope attributes. (#5934) - `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` adds instrumentation scope attributes. (#5934) - `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc` adds instrumentation scope attributes. (#5935) - `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp` adds instrumentation scope attributes. (#5935) - `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc` adds instrumentation scope attributes. (#5933) - `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp` adds instrumentation scope attributes. (#5933) - `go.opentelemetry.io/otel/exporters/prometheus` adds instrumentation scope attributes in `otel_scope_info` metric as labels. (#5932) ### Changed - Support scope attributes and make them as identifying for `Tracer` in `go.opentelemetry.io/otel` and `go.opentelemetry.io/otel/sdk/trace`. (#5924) - Support scope attributes and make them as identifying for `Meter` in `go.opentelemetry.io/otel` and `go.opentelemetry.io/otel/sdk/metric`. (#5926) - Support scope attributes and make them as identifying for `Logger` in `go.opentelemetry.io/otel` and `go.opentelemetry.io/otel/sdk/log`. (#5925) - Make schema URL and scope attributes as identifying for `Tracer` in `go.opentelemetry.io/otel/bridge/opentracing`. (#5931) - Clear unneeded slice elements to allow GC to collect the objects in `go.opentelemetry.io/otel/sdk/metric` and `go.opentelemetry.io/otel/sdk/trace`. (#5804) ### Fixed - Global MeterProvider registration unwraps global instrument Observers, the undocumented Unwrap() methods are now private. (#5881) - `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc` now keeps the metadata already present in the context when `WithHeaders` is used. (#5892) - `go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc` now keeps the metadata already present in the context when `WithHeaders` is used. (#5911) - `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc` now keeps the metadata already present in the context when `WithHeaders` is used. (#5915) - Fix `go.opentelemetry.io/otel/exporters/prometheus` trying to add exemplars to Gauge metrics, which is unsupported. (#5912) - Fix `WithEndpointURL` to always use a secure connection when an https URL is passed in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc`. (#5944) - Fix `WithEndpointURL` to always use a secure connection when an https URL is passed in `go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp`. (#5944) - Fix `WithEndpointURL` to always use a secure connection when an https URL is passed in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc`. (#5944) - Fix `WithEndpointURL` to always use a secure connection when an https URL is passed in `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp`. (#5944) - Fix incorrect metrics generated from callbacks when multiple readers are used in `go.opentelemetry.io/otel/sdk/metric`. (#5900) ### Removed - Remove all examples under `go.opentelemetry.io/otel/example` as they are moved to [Contrib repository](https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/examples). (#5930)

main: add test case to reproduce the behaviour, it will fail as of no…

04a0ff9

…w, once the fix goes it will pass

pree-dew requested review from MrAlias, XSAM, dashpole, pellared and dmathieu as code owners October 19, 2024 09:52

dmathieu marked this pull request as draft October 19, 2024 11:15

dashpole added this to the v1.32.0 milestone Oct 21, 2024

dashpole added bug Something isn't working area:metrics Part of OpenTelemetry Metrics labels Oct 21, 2024

dashpole self-assigned this Oct 21, 2024

pree-dew added 2 commits October 28, 2024 22:00

Merge branch 'main' into 5866-multiple-readers-producing-incorrect-me…

b015b16

…trics

1. Separate observer for each pipeline 2. Register measure for each p…

3d2c60a

…ipeline and get the measure during collect for the called pipeline

pree-dew mentioned this pull request Oct 28, 2024

Multiple periodic readers increment the counter abnormaly #5866

Closed

dashpole reviewed Oct 28, 2024

View reviewed changes

sdk/metric/meter.go Outdated Show resolved Hide resolved

sdk/metric/pipeline.go Outdated Show resolved Hide resolved

sdk/metric/meter.go Show resolved Hide resolved

sdk/metric/meter.go Show resolved Hide resolved

1. Add test case for add measures. 2. Avoid running validation of ins…

bce8a45

…trument for each pipeline. 3. remove the registercallback function

dashpole reviewed Nov 1, 2024

View reviewed changes

sdk/metric/meter.go Outdated Show resolved Hide resolved

sdk/metric/meter.go Show resolved Hide resolved

sdk/metric/pipeline.go Outdated Show resolved Hide resolved

1. check if register is required or not via validInstruments. 2. Add …

eb3f053

…test case to check concurrency on read and write of observer. 3. Remove the check of adding measure as it will get response from cache.

pree-dew added 2 commits November 1, 2024 22:51

add Changelog.md for the issue

26387ea

merge with main

8275538

dashpole reviewed Nov 1, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

add entry to fixed section and change the log for better user underst…

673c4ef

…anding

pree-dew changed the title ~~[WIP] Fix incorrect metrics getting generated from multiple readers~~ Fix incorrect metrics getting generated from multiple readers Nov 1, 2024

pree-dew marked this pull request as ready for review November 1, 2024 18:20

dashpole approved these changes Nov 1, 2024

View reviewed changes

dmathieu approved these changes Nov 4, 2024

View reviewed changes

pellared reviewed Nov 6, 2024

View reviewed changes

sdk/metric/pipeline_test.go Outdated Show resolved Hide resolved

sdk/metric/pipeline_test.go Outdated Show resolved Hide resolved

sdk/metric/pipeline_test.go Outdated Show resolved Hide resolved

sdk/metric/pipeline_test.go Outdated Show resolved Hide resolved

pree-dew added 2 commits November 6, 2024 20:35

Merge branch 'main' into 5866-multiple-readers-producing-incorrect-me…

49d12f3

…trics

1. Change reader duration. 2. Change test name. 3. Change the assert …

6558cd2

…mechanism

pellared reviewed Nov 7, 2024

View reviewed changes

sdk/metric/pipeline_test.go Outdated Show resolved Hide resolved

pellared reviewed Nov 7, 2024

View reviewed changes

sdk/metric/pipeline_test.go Outdated Show resolved Hide resolved

pellared reviewed Nov 7, 2024

View reviewed changes

sdk/metric/pipeline_test.go Outdated Show resolved Hide resolved

1. Remove periodic reader from test to avoid any time based logic

fcfa966

pellared approved these changes Nov 7, 2024

View reviewed changes

pellared mentioned this pull request Nov 7, 2024

sdk/metric: Refactor pipeline and observable measures #5946

Open

Add comments

3b96d69

pellared added the pkg:SDK Related to an SDK package label Nov 7, 2024

XSAM approved these changes Nov 8, 2024

View reviewed changes

Merge branch 'main' into 5866-multiple-readers-producing-incorrect-me…

536497b

…trics

pellared merged commit 1492efa into open-telemetry:main Nov 8, 2024
31 checks passed

pellared mentioned this pull request Nov 8, 2024

Release v1.32.0/v0.54.0/v0.8.0/v0.0.11 #5960

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix incorrect metrics getting generated from multiple readers #5900

Fix incorrect metrics getting generated from multiple readers #5900

pree-dew commented Oct 19, 2024 •

edited

Loading

dashpole left a comment

codecov bot commented Nov 1, 2024 •

edited

Loading

dashpole commented Nov 1, 2024

pree-dew commented Nov 1, 2024 •

edited

Loading

dashpole commented Nov 1, 2024

dashpole left a comment

pellared commented Nov 7, 2024

pellared commented Nov 8, 2024

pree-dew commented Nov 8, 2024

Fix incorrect metrics getting generated from multiple readers #5900

Fix incorrect metrics getting generated from multiple readers #5900

Conversation

pree-dew commented Oct 19, 2024 • edited Loading

dashpole left a comment

Choose a reason for hiding this comment

codecov bot commented Nov 1, 2024 • edited Loading

Codecov Report

dashpole commented Nov 1, 2024

pree-dew commented Nov 1, 2024 • edited Loading

dashpole commented Nov 1, 2024

dashpole left a comment

Choose a reason for hiding this comment

pellared commented Nov 7, 2024

pellared commented Nov 8, 2024

pree-dew commented Nov 8, 2024

pree-dew commented Oct 19, 2024 •

edited

Loading

codecov bot commented Nov 1, 2024 •

edited

Loading

pree-dew commented Nov 1, 2024 •

edited

Loading