Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: write public stats to our PG DB #122

Merged
merged 11 commits into from
Jan 16, 2024
Merged

feat: write public stats to our PG DB #122

merged 11 commits into from
Jan 16, 2024

Conversation

bajtos
Copy link
Member

@bajtos bajtos commented Jan 11, 2024

A prototype implementing code for producing stats to power our Spark Public Dashboard. Design doc:
https://pl-strflt.notion.site/DB-for-SPARK-public-data-a79cbb7a0ff5447980b714afdb2645d6

  1. Add a runtime dependency on a PG database, using a new database
    name spark_stats (originally: spark_public). Setup database schema migrations executed
    at service startup.

  2. Define the first table retrievals to keep stats about all vs
    succesfull retrievals performed each day.

  3. Update the evaluate step to update these stats for each round
    completed.

TODO

  • Setup PG DB for GitHub Actions to make the tests pass
  • Create a new PG user spark_evaluate and a new database spark_stats
  • Define DATABASE_URL in Fly.io secrets

Links:

1. Add a runtime dependency on a PG database, using a new database
   name `spark_public`. Setup database schema migrations executed
   at service startup.

2. Define the first table `retrievals` to keep stats about all vs
   succesfull retrievals performed each day.

3. Update the evaluate step to update these stats for each round
   completed.

Signed-off-by: Miroslav Bajtoš <[email protected]>
bin/spark-evaluate.js Outdated Show resolved Hide resolved
migrations/001.do.retrieval-stats.sql Outdated Show resolved Hide resolved
bin/spark-evaluate.js Outdated Show resolved Hide resolved
@bajtos bajtos marked this pull request as ready for review January 15, 2024 12:31
@bajtos bajtos requested a review from juliangruber January 15, 2024 12:32
Signed-off-by: Miroslav Bajtoš <[email protected]>
Avoid reading process.env.DATABASE_URL deep inside the call stack.

Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
Copy link
Member Author

bajtos commented Jan 15, 2024

Last minute change: let's rename the DB from spark_public to spark_stats. It better captures what we are storing. In the future, we can store in this DB data for our private dashboards too.

Signed-off-by: Miroslav Bajtoš <[email protected]>
Copy link
Member

@juliangruber juliangruber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work!

@bajtos bajtos merged commit e934583 into main Jan 16, 2024
5 checks passed
@bajtos bajtos deleted the public-data branch January 16, 2024 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: ✅ done
Development

Successfully merging this pull request may close these issues.

2 participants