ETL superset usage metrics

[bendnorman]

ETL Superset logs so we can understand who is using it and how!

There are a few sources of superset information

1. the superset database and
2. the Cloud logs produced by Cloud Run.
3. the Cloud Monitoring Metrics of the Cloud Run instance.

I think the database will provide valuable information like users, queries and logs. This comment has some more information. The Cloud Logs will likely have everything in the logs table plus some additional Cloud Run operational information which could be helpful for monitoring the performance of specific operations. The Monitoring metrics information would be helpful for monitoring overall performance of the deployment. Monitoring metrics are currently retained for 6 weeks so we'd need to archive them.

I could see a few different phases for this work:

Phase I

• Connect the superset db to superset and build a few simple dashboards off the raw tables.

Phase II

• Set up a Cloud Log Sink for Cloud run so we can archive the logs Create a Cloud Logging sink for superset logs #178
• Archive the Cloud monitoring metrics some how
• If we feel like if the superset database tables require additional processing we could add it to the ETL

Phase II

• Incorporate the Cloud Logs and Monitoring metrics into our ETL and dashboard if we think they are useful

Next steps

Beta Give feedback

1. ...

[bendnorman]

Also, @e-belfer I connected the superset database to superset so you can explore the data.