chore: Distinction between gross/net trafic in bandwidth per shard metric, a…

[NagyZoltanPeter]

...added bandwidths and request rate panels to single node and fleet dashboards

Description

This was a leftover task after @Ivansete-status comments on #2851
Now gross and net traffic of incoming per shard relay measured separately.
Gross traffic means all incoming relay traffic (measured before validation). It will help us to understand how much of the bandwidth stress on the node vs. useful relay traffic that is actually valid and propagated forward. This is now measured per shard.

Changes

• Introduced new counter filter of "type" for waku_relay_network_bytes metric, that represents distinction of gross/net traffic
• In connection to feat: Added proper per shard bandwidth metric calculation #2851 I pulled and adjusted metrics panels to single node and fleet dashboards.
  • Added waku relay traffic per shads in/out
  • Added request rate, non relay traffic in/out separately in respective rows of store, filter and lightpush

Unfortunately, I cannot create pictures of working dashboards as it is not simple to try out the changes on fleet dashboard.

Issue

#1945

Connected PR

nwaku-compose respective changes: waku-org/nwaku-compose#99

[github-actions]

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2920

Built from 70def48

[gabrielmer]

Thank you! Could you please add a screenshot of the new dashboard?

[NagyZoltanPeter]

Thank you! Could you please add a screenshot of the new dashboard?

@gabrielmer : Actually these dashboards, specially the fleet one needs special environment.
But if you have an idea how to run grafana that way to show meaningful data on those panels... I will be thankful.

[gabrielmer]

@gabrielmer : Actually these dashboards, specially the fleet one needs special environment. But if you have an idea how to run grafana that way to show meaningful data on those panels... I will be thankful.

Not sure I understand then, so how did these files get generated? I thought you created the dashboards in Grafana and update the files with the ones of the dashboard you built

[NagyZoltanPeter]

@gabrielmer : Actually these dashboards, specially the fleet one needs special environment. But if you have an idea how to run grafana that way to show meaningful data on those panels... I will be thankful.

Not sure I understand then, so how did these files get generated? I thought you created the dashboards in Grafana and update the files with the ones of the dashboard you built

Well I was using nwaku-compose, I put them there, but prometheus under nwaku-compose do not provide the right collection to populate the dashboard with data. Like for fleet we have datacenter, node type and other distinctions, but for that you would need to setup properly the whole, which I did not. I was copying from nwaku-compose's dashboard the panels and adapt the queries. Kinda' hack.

[gabrielmer]

Well I was using nwaku-compose, I put them there, but prometheus under nwaku-compose do not provide the right collection to populate the dashboard with data. Like for fleet we have datacenter, node type and other distinctions, but for that you would need to setup properly the whole, which I did not. I was copying from nwaku-compose's dashboard the panels and adapt the queries. Kinda' hack.

Got it! Yeah I'm also not sure how/where these panels are supposed to be run, I didn't know they existed.

@Ivansete-status do you know where are they used and how they can be tested? Are they specifically for https://grafana.infra.status.im/ ?

[Ivansete-status]

LGTM! Thanks for it! 💯
If possible it'd help to have screenshot of new panels.
I wonder though how can we deploy the new dashboards into the fleet's Grafana

[gabrielmer]

Approving also not to block with the monitoring part :)) Thank you!