Improved Status Monitoring

[lbeckman314]

Currently the status monitor only reflects whether the sites are up and reachable. It would be beneficial to expand the monitoring to include individual services and project status:

Service Status

• Status of individual services (i.e. is Fence/Arborist/Sheepdog/etc. up?)
  • Include posting to the aced-status-monitor Slack channel if a service goes down:
    • Example message: "Staging: Fence service is down"
    • Could be based on 'livenessProbe' already present in the deployments:
      helm/fence/templates/fence-deployment.yaml

    livenessProbe:
      httpGet:
        path: /_status
        port: http

ETL/data status

• Include posting to the aced-status-monitor Slack channel when data is uploaded:
  • Example message: "Staging ETL: Data added to <project>", "Staging ETL: Errors in data from <submitter>"

Text Messages/E-mail Support

• Additionally it would be nice to add support for simple text messages and e-mail notifications via Upptime to allow developers to opt-in to their preferred mode of communication.