[Deliverable] Metric: Count store messages

[chair28980]

Project: https://github.com/orgs/waku-org/projects/22/views/1

Description

Message-finder is used to compare the number of Status messages across Status nodes to understand the potential discrepancies and odd behaviour of messages being inserted in the past in a given channel.

This deliverable captures the message count on all Waku fleets to use as a metrics of the efficacy of store sync.

Tool is being created in https://github.com/waku-org/storenode-messages repo, work on other clients is not expected.

Tasks

• Implement storenode message counter storenode-messages-counter#8

Output

https://grafana.infra.status.im/d/hmX6pJ_Iz/storenode-message-counter?orgId=1&from=now-7d&to=now

[chair28980]

Richard is creating a separate tool for this task, work on other clients not expected.

waku-org/storenode-messages-counter#8

[chair28980]

Closing as completed waku-org/storenode-messages-counter#8 (comment)

[fryorcraken]

Re-opened because we need some graph or dashboard to actually see the missing message count, so that we can check that with store sync there are indeed no recent message missing.

There was also performance issue due to the traffic spike.

@richard-ramos Are performance improvement done? When do you think we can have the graphs?

[richard-ramos]

Graphs can be seen here: https://grafana.infra.status.im/d/hmX6pJ_Iz/storenode-message-counter?orgId=1

[richard-ramos]

I did add some degree of concurrency to try improve the performance. So far the delay to process the messages seem to hover around ~8m which seems fine

[fryorcraken]

Latest update on this. The current diagram are good to highlight issues with the store node. However, it does not enable us to confirm whether store sync are able to fix missing messages. The proposal is to add a secondary check that reviews if known missing messages are still missing after some times. Here is the description of the graphs:

3 graphs per period:

• past hour
• past day
• past week

for each

• x axis is time of recheck
• number of still missing messages within the period of the graph (per store node)

So if you look at past hour, it shows at time t (time of recheck), what messages where still missing within t - 1h to ~t or most likelyt - 2 min

So ideally, from the moment store sync is enabled, past day and past week always flat line.

And past hour can be used to debug store sync and improve it to flat line it.

And this way, we can also affirm that the store are sync'd over given period of time.

@SionoiS @Ivansete-status if you can confirm that such change would help confirm that store sync is able to recover store node message discrepancies that would be great.

Also please clarify:

• how often does/will store sync run?
• what period of time does store sync covers (how much in the past it tries to sync messages)

on Status fleets.

[fryorcraken]

Also, @richard-ramos, can we also deploy it for TWN? Would be good to have if projects complain of message loss. not urgent.

[SionoiS]

• how often does/will store sync run?

Every hours.

• what period of time does store sync covers (how much in the past it tries to sync messages)

Last hour minus 20s.

[Ivansete-status]

@fryorcraken

@SionoiS @Ivansete-status if you can confirm that such change would help confirm that store sync is able to recover store node message discrepancies that would be great.

I think this is a great idea but we can make it simpler, i.e., make the storenodes-messages tool to keep track of the number of seen messages on each storenode, and then just represent the relative difference amongst them. wdyt @richard-ramos ?

[richard-ramos]

Hm. I did add the message re-verification requested by Franck earlier and did not add much complexity to the code. I think it should be fine since it is extra information that might be useful. If later we see that it is not really needed, we can drop the changes (I'm all in for removing code! :) )

[fryorcraken]

All features done and added to dashboard: https://grafana.infra.status.im/d/hmX6pJ_Iz/storenode-message-counter?orgId=1&from=now-7d&to=now

Dogfooding started. Will review in a week and close deliverable.

[fryorcraken]

Done!