Public Spark Dashboard MVP

[patrickwoodhead]

The funders of Spark would like to have a public dashboard so that they can see the results of Spark.

Currently, the Spark dashboard is on Grafana behind an auth wall. This means only a few people can see the data. We could make this dashboard public but this would open up our database to public internet requests. We need to find a solution whereby the dashboard is public but the database isn't open to the world.

Product Requirements

• one data point each day and keep this data with unlimited retention, for each chart
• use Grafana for the UI

Tasks

• Architecture plan for public dashboard
• Chart that shows retrieval success rate
• Chart that shows RSR with a filter by Storage Provider
• Aggregate stats from round evaluations
• Expose stats via REST API
• Average RSR of non zero RSR SPs
• Deals Checked 1d/7d/30d (formerly: CIDs sampled)
• Retrievals performed 1d/7d/30d

Stretch goals

• Add more columns to the miner RSR table
  • number of measurements performed
  • breakdown of common errors (not advertising to IPNI, retrieval failed with 502, etc.)
• Add a chart to show historic change in success rate for a chosen provider

Moved to #120

Out of scope

• Measure the fraction of Filecoin we are checking filecoin-station/spark#34
• Build a Public Spark Dashboard in our new Grafana account #72
• Measure and visualise requests per storage provider #64

[bajtos]

Related issues - potential future ideas:

• Measure the fraction of Filecoin we are checking filecoin-station/spark#34

[bajtos]

Let's aggregate raw data to produce one data point each day and keep this data with unlimited retention.

[bajtos]

Retrieval Bot's dashboard for inspiration:

https://retrievalbot-dashboard.vercel.app

[bajtos]

Dashboard PoC: https://bajtos.grafana.net/public-dashboards/9af156c3266d4d8384385623d0d2a76c

[bajtos]

Public dashboard URL - redirect is configured via Cloudflare:

https://filspark.com/dashboard