Aggregrated stats view

[parthshahva]

This ticket is a spike. Notes need to be cleaned up and turned into a checklist of deliverables

As an Operations person, I what to know what know at a high level what the DCP is doing and how busy each component is, so that I can spot components under unusual strain, which may indicate a system problem or ongoing system abuse.

I’m going to ramble here for a bit while I figure out what #9 means:
What does “busy” mean for each component?
Ingest is difficult: you can’t use “number submissions in progress” as they may take months to be completed.
Upload
number of files uploaded in the last hour?
Total number of files currently stored
Checksums running
Validations running
Data Store - files/bundles stored in the last hour?
Secondary analysis is easy - how many workflows are running.

From f2f session
Ingest -
Busy? queue for requests to export bundles, expose linking progress in spreadsheet processing
Something went wrong? High failed export bundle counts or large # of envelopes stuck in pre-completed phases
Upload -
Busy? # of files pending validation, # in progress, # failed
Something went wrong? Large # of failed validations
DSS
Busy? Bundles created, syncing of data, notification queue
Something went wrong? Large # of data failing to sync to failed notifications
Analysis?
Busy? Running and Pending workflows
Something went wrong? Large # of recently failed or hung workflows
Azul
Busy? Currently indexing bundles or queue responding to notifications from dss
Something went wrong? Large # of failed notification processing failures