Telemetry support #169
Merged
Telemetry support #169
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* WIP: Starting bootstrapping prometheus * Minor refactor * Fix init condition * Add libp2p metrics * Add echo metrics * WIP: metrics rename * Fix closing of libp2p host * Use milliseconds for time resolution + use prefixes for different packages * Use variables for metric names for node * Fstore and host also use vars for metrics * Add descriptions to defined metrics * Add more metrics - consensus time and messages sent/published * Add node info to metrics * Obsolete comment * Minor tweaks * (re)Add node ID to the node_info metric
* Update/fix tests * Refactor telemetry init functions * Tests for resource and simple tracing (in mem exporter) * Add test for tracing wrapped functions * Add tests for trace info * Decouple prometheus definitions bootstrapping by using config * Remove printf * Add test to trace health check and worker execution * Go mod tidy * Offload pipeline type to a separate package * Telemetry tests are external * Refactor metric initialization and start adding tests for metrics * Add tests for metrics config * Use metrics as a subcomponent instead of the global instance * Rename receiver in fstore code * Add test for processed messages * Rename test file * Add metrics verification to integration test * Split tracing and metrics initialization * Use new context for untraced test * Remove addressed comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces support for telemetry for
b7snodes.We have support for two main mechanisms - tracing and metrics.
Tracing
Tracing can be enabled with the CLI flags
--enable-tracingand either--tracing-grpc-endpointor--tracing-http-endpointflags, depending on which protocol should be used to transfer tracing data.Below is an example configuration for a config file:
Some more background about the implementation can be found here - #158 .
Metrics
Metrics can be enabled with the CLI flags
--enable-metricsand--prometheus-address. The latter CLI flag controls on which address the worker node will serve the metrics. In the case of the head node, the metrics will be served on the REST API address.Metrics can be found on the
/metricsendpoint.Support for push metrics was dropped because it seems against the Prometheus guidelines, and pull metrics can be enabled by using Prometheus PushProx.
For the config file, the metrics configuration might be:
More information about the implementation can be found here - #164 .
Note that these two PRs are not the final version, and some changes were made afterwards, here - #167 ,
and in minor scale in this PR, after rebasing the feature branch.
Subsequent changes mainly deal with splitting
tracingandmetricsand supporting enabling one or the other, instead of forcingtelemetryas a single system.