chore: extending store metrics

[Ivansete-status]

Description

Brings new metrics to analyse the time spent by store.
In a store request, we have these time components:

1. database-time: time spent by the query itself within the database.
2. communication-with-database: time spent by libpq while waiting for response.
3. store-lpstream-resp-time: time spent while returning the response to the store-client.

We will have stats for (1 + 2) and (3)

From nwaku we don't have means to measure the database time, which should be manually measured by analysing pg_stat_statemets table with queries like SELECT calls, total_exec_time, mean_exec_time, max_exec_time, left(query, 80) rows FROM pg_stat_statements ORDER BY total_exec_time DESC LIMIT 100;.

Changes

• New store and database metrics are added
• New grafana panels aiming to better describe query time consumption
  

Issue

• closes chore: create grafana panel to show max, min, mean, median stats of query times #2995

[github-actions]

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:3042

Built from 3c63460

[gabrielmer]

Looks amazing! 🔥 thanks so much!

I think there's a couple linting changes in libwaku. Not sure if they're intentional or not as they seem to be unrelated to this

[NagyZoltanPeter]

Very insightful!
But what is different between the two "Waku Store Time (sec)" panels?
It is not obvious from the picture.
Also maybe the panels can get more height for better legend readability.

[Ivansete-status]

I think there's a couple linting changes in libwaku. Not sure if they're intentional or not as they seem to be unrelated to this

Thanks for the comment! ah yes, the thing is that I wasn't properly using the nph-0.6.0. They are intentional but aside from this PR ;P

[Ivansete-status]

Very insightful! But what is different between the two "Waku Store Time (sec)" panels? It is not obvious from the picture. Also maybe the panels can get more height for better legend readability.

Thanks! I've extended the panels a little so that we can see better the queries and also made it clearer the "Store Times" panels, which will show legacy and new store times distribution, for example, we have this now:

[NagyZoltanPeter]

Yess.
Thank you It looks just fine now!