-
Notifications
You must be signed in to change notification settings - Fork 597
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Incorrect StartTimeUnixNano for ExponentialHistogram with Delta Temporality after gRPC connection drop #3974
Comments
I've performed more testing and it seems the gRPC disconnection is irrelevant; the aged StartTimeUnixNano is used regardless. |
@alexchowle care to give #3978 a run? |
I've ran the reproduction scenario and am still seeing the same issue.
I'm pretty confident I'm using the correct I'm thinking I'm doing something wrong with my builds/imports of the wheels. |
From the data above you are making 2 observations, with the following timestamps:
This is expected, the start time of a DELTA temporality collected exponential histogram data point is the value of the collection time of the previous DELTA temporality collected exponential histogram data point. For this behavior to change to match the Golang behavior the entire I'll ask @jmacd for guidance here. |
I only noticed this issue because my vendor backend uses |
Sorry, please disregard my previous answer I need to do more thinking about this issue, and consider the export interval of the periodic exporter, will comment here when I have a better answer ✌️ |
@alexchowle I have created a branch that I think should fix your issue: #4002 I am just adding a commit so that the previous collection time gets updated with every collection regardless of Please give it a try ✌️ |
I disagree. The supplementary guidelines has a delta example which I believe covers this situation. Notice how the start / end time of the 3rd export is |
Please take a look here |
I've just tried it. This looks very promising: Observation # 1
Observation # 2
|
Describe your environment
OS: CentOS 7
Python version: 3.11.1
SDK version: 1.25.0
API version: 1.25.0
What happened?
As part of investigating #3971 I found that the loss and subsequent re-establishment of a gRPC connection to a Collector caused the next metric export to have a very old StartTimeUnixNano value
Steps to Reproduce
Have the OTel Collector drop the client's gRPC connection by setting the
max_connection_age
to5m
(as documented at https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configgrpc/README.md#server-configuration)So the first recording happens, then within the 6-minute sleep the Collector drops the client's gRPC connection within 5 minutes, then the client sends a second observation after the 6 minutes has elapsed.
Expected Result
StartTimeUnixNano has a value that is within 60 seconds of the 2nd recording. This is how the Golang SDK works.
Actual Result
StartTimeUnixNano has the TimeUnixNano of observation # 1 i.e. about 6 minutes in the past.
Additional context
No response
Would you like to implement a fix?
None
The text was updated successfully, but these errors were encountered: