Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding custom labels to metrics #222

Open
rajyan opened this issue Aug 2, 2023 · 9 comments · May be fixed by #291
Open

Adding custom labels to metrics #222

rajyan opened this issue Aug 2, 2023 · 9 comments · May be fixed by #291
Assignees

Comments

@rajyan
Copy link

rajyan commented Aug 2, 2023

Hi, thank you for this great library.

I was wondering if there is a way to add custom labels to client metrics.
I can add default attributes with otelsql.WithDefaultAttributes, but is it possible to add semconv.DBStatementKey.String or query arguments for labels?

@rajyan
Copy link
Author

rajyan commented Aug 3, 2023

Some kind of middleware like

otelsql/options.go

Lines 157 to 161 in f4a240f

func TraceQuery(f queryTracer) DriverOption {
return driverOptionFunc(func(o *driverOptions) {
o.trace.queryTracer = f
})
}

for metrics might be reasonable?

@nhatthm
Copy link
Owner

nhatthm commented Aug 3, 2023

What is your use case exactly? 🤔

@rajyan
Copy link
Author

rajyan commented Aug 3, 2023

What is your use case exactly? 🤔

I was thinking about

  • Count, sum up duration by DBStatementKey to look for the performance by query
  • Add labels from context, like http.route to see the stats by other contexts

@nhatthm
Copy link
Owner

nhatthm commented Aug 3, 2023

Putting query and its args into the metric labels will drastically increase the cardinality of metrics. You should go with one of these alternatives:

  1. Use Traces instead of Metrics. Even with a low sampling rate, it still provides insightful data about the performance of the operations. Some platforms, such as Grafana Tempo or Honeycomb, provide convenience to filter the spans and attributes.
  2. Measure the operations at one higher level, for example, the place that calls db.QueryContext(). That will provide the metrics with better context, rather than just the query and the args.

@rajyan
Copy link
Author

rajyan commented Aug 3, 2023

Putting query and its args

To be clear, I was planning to add query without args, which I think the cardinality should be reasonable (at least in my case).
For args I was thinking about more filtered usage, for example, adding labels by specific args that has low cardinality.

Thank you for the alternatives. I'm currently using Grafana Tempo for traces and using Prometheus for metrics.
Let me be even more specific, I'm trying to create a query dashboard like AWS RDS Performance Insights
https://aws.amazon.com/jp/blogs/database/amazon-rds-performance-insights-is-now-generally-available/
but couldn't find a way to do so. This is why I created this issue (I'm sorry if this is the wrong way to go).

@rajyan
Copy link
Author

rajyan commented Aug 3, 2023

Searching for Grafana Tempo,
https://community.grafana.com/t/dashboard-with-arithmetic-operation-between-two-spans-of-a-trace/88794

For this use-case, the best option is using the metrics-generator 1 with span-metrics 2. This will create latency metrics of your spans, to which you can add attributes that are relevant for you, and then calculate the difference in latency.

maybe this can work in my use case?

@rajyan
Copy link
Author

rajyan commented Aug 13, 2023

@nhatthm

Created a sample query dashboard in Grafana that can be achieved by this feature.
Hope this make sense.

image

Also, I'm interested in implementing this if you think it's a good idea

@nhatthm
Copy link
Owner

nhatthm commented Aug 14, 2023

Searching for Grafana Tempo, https://community.grafana.com/t/dashboard-with-arithmetic-operation-between-two-spans-of-a-trace/88794

For this use-case, the best option is using the metrics-generator 1 with span-metrics 2. This will create latency metrics of your spans, to which you can add attributes that are relevant for you, and then calculate the difference in latency.

maybe this can work in my use case?

I think you can try that.


Regarding the original question, this library isn't the right tool for building dashboards like AWS Performance Insights. You will need an exporter that gathers data directly from the database servers. Try to search for XXX prometheus exporter, e.g Postgres exporter.

@rajyan
Copy link
Author

rajyan commented Aug 14, 2023

Thank you for you for your suggestions.

I think you can try that.

I was able to achieve something similar using this method as well, just to let you know.
Although, it is not as flexible as setting those attributes in otelsql.

Regarding the original question, this library isn't the right tool for building dashboards like AWS Performance Insights.

Maybe I'm not understanding the concept of OpenTelemetry metrics correctly, but what is the supposed use case of the otelsql metrics then?

The differences that I think collecting data from otelsql instead of prometheus is that

  • The metrics table in Adding custom labels to metrics #222 (comment) has links to traces
  • The metrics has contexts that can only obtained from application. Same queries might be called in different endpoints, and we can add those information
  • The observations won't depend on the database driver. We can switch from MySQL to PostgreSQL easily

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants