Invalid loki alert rules fail silently

[cbartz]

Bug Description

If you define an invalid loki alert rule, it will simply not be transmitted without error logs, causing unnecessary troubleshooting time.

I would expect the Grafana agent to provide error logs.

To Reproduce

1. Define an invalid loki alert rule file for your charm.

e.g.:

groups:
  - name: github-runner-capacity
    rules:
      - alert: Job Queue duration too high
        expr: |
          quantile_over_time(0.5, {filename="/var/log/github-runner-metrics.log"} | json event="event",duration="queue_duration",flavor="flavor" | __error__="" | event="runner_start" | unwrap duration[1h]) by(flavor) > 1800
        labels:
            severity: high
        for: 1h
        annotations:
            description: "Job queue duration is higher than 30 minutes for half of the runners with flavor {{$labels.flavor}}"
            summary: "Job queue duration is too high for flavor {{$labels.flavor}}
      - alert: No idle runners
        expr: |
          sum by(flavor)(last_over_time({filename="/var/log/github-runner-metrics.log"} | json event="event",idle_runners="idle_runners",flavor="flavor" | event="reconciliation" | unwrap idle_runners[1h])) == 0
        labels:
            severity: high
        for: 2h
        annotations:
            summary: "No idle runners for flavor {{$labels.flavor}}"
      - alert: Short idle time
        expr: |
          avg_over_time({filename="/var/log/github-runner-metrics.log"} | json event="event",idle="idle",flavor="flavor" | event="runner_start" | unwrap idle[1h]) by(flavor) < 300
        labels:
            severity: high
        for: 2h
        annotations:
            description: "Idle time for flavor {{$labels.flavor}} is on average smaller than 5 minutes."
            summary: "Idle time for flavor {{$labels.flavor}} is too short"

see missing quote after "Job queue duration is too high for flavor {{$labels.flavor}}
3. Integrate with grafana-agent.
4. Observe that all the rules in the file are not transmitted and that there are no errors in the juju debug logs.

Environment

grafana agent latest code on main (should correspond to rev 144)

Relevant log output

unit-grafana-agent-66: 14:46:39 INFO juju.worker.apicaller [eb824e] "unit-grafana-agent-66" successfully connected to "10.158.39.89:17070"
unit-grafana-agent-66: 14:46:39 INFO juju.worker.apicaller [eb824e] password changed for "unit-grafana-agent-66"
unit-grafana-agent-66: 14:46:39 INFO juju.worker.apicaller [eb824e] "unit-grafana-agent-66" successfully connected to "10.158.39.89:17070"
unit-grafana-agent-66: 14:46:39 INFO juju.worker.migrationminion migration phase is now: NONE
unit-grafana-agent-66: 14:46:39 INFO juju.worker.logger logger worker started
unit-grafana-agent-66: 14:46:39 INFO juju.worker.upgrader no waiter, upgrader is done
unit-grafana-agent-66: 14:46:39 INFO juju.worker.uniter unit "grafana-agent/66" started
unit-grafana-agent-66: 14:46:39 INFO juju.worker.uniter resuming charm install
unit-grafana-agent-66: 14:46:39 INFO juju.worker.uniter.charm downloading local:jammy/grafana-agent-2 from API server
unit-grafana-agent-66: 14:46:40 ERROR juju.worker.meterstatus error running "meter-status-changed": charm missing from disk
unit-grafana-agent-66: 14:46:39 INFO juju Starting unit workers for "grafana-agent/66"
unit-grafana-agent-66: 14:47:10 INFO juju.worker.uniter hooks are retried true
unit-grafana-agent-66: 14:47:10 INFO juju.worker.uniter.storage initial storage attachments ready
unit-grafana-agent-66: 14:47:10 INFO juju.worker.uniter found queued "install" hook
unit-grafana-agent-66: 14:47:10 DEBUG unit.grafana-agent/66.juju-log ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:10 INFO unit.grafana-agent/66.juju-log Running legacy hooks/install.
unit-grafana-agent-66: 14:47:11 DEBUG unit.grafana-agent/66.juju-log ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:11 DEBUG unit.grafana-agent/66.juju-log Charm called itself via hooks/install.
unit-grafana-agent-66: 14:47:11 DEBUG unit.grafana-agent/66.juju-log Legacy hooks/install exited with status 0.
unit-grafana-agent-66: 14:47:11 DEBUG unit.grafana-agent/66.juju-log Using local storage: not a Kubernetes podspec charm
unit-grafana-agent-66: 14:47:11 DEBUG unit.grafana-agent/66.juju-log Initializing SQLite local storage: /var/lib/juju/agents/unit-grafana-agent-66/charm/.unit-state.db.
unit-grafana-agent-66: 14:47:11 DEBUG unit.grafana-agent/66.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules: directory does not exist
unit-grafana-agent-66: 14:47:11 DEBUG unit.grafana-agent/66.juju-log no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:11 DEBUG unit.grafana-agent/66.juju-log <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f68692ba430> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:11 DEBUG unit.grafana-agent/66.juju-log Emitting Juju event install.
unit-grafana-agent-66: 14:47:11 INFO unit.grafana-agent/66.juju-log Installing snap grafana-agent, revision None, tracking 
unit-grafana-agent-66: 14:47:16 INFO unit.grafana-agent/66.juju-log The snap installation completed successfully
unit-grafana-agent-66: 14:47:16 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:17 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:17 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules: directory does not exist
unit-grafana-agent-66: 14:47:17 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:17 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f8cf2160e50> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:17 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Emitting Juju event cos_agent_relation_created.
unit-grafana-agent-66: 14:47:18 INFO juju.worker.uniter.operation ran "cos-agent-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:19 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:19 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules: directory does not exist
unit-grafana-agent-66: 14:47:19 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:19 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f28462e8ea0> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:19 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Emitting Juju event send_remote_write_relation_created.
unit-grafana-agent-66: 14:47:19 INFO juju.worker.uniter.operation ran "send-remote-write-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:20 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:20 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules: directory does not exist
unit-grafana-agent-66: 14:47:20 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:20 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7fc6ff4c8cc0> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:20 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: Emitting Juju event grafana_dashboards_provider_relation_created.
unit-grafana-agent-66: 14:47:21 INFO juju.worker.uniter.operation ran "grafana-dashboards-provider-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:21 DEBUG unit.grafana-agent/66.juju-log peers:119: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:21 DEBUG unit.grafana-agent/66.juju-log peers:119: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules: directory does not exist
unit-grafana-agent-66: 14:47:21 DEBUG unit.grafana-agent/66.juju-log peers:119: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:21 DEBUG unit.grafana-agent/66.juju-log peers:119: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f0c00d38d10> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:21 DEBUG unit.grafana-agent/66.juju-log peers:119: Emitting Juju event peers_relation_created.
unit-grafana-agent-66: 14:47:21 INFO unit.grafana-agent/66.juju-log peers:119: certhandler waiting on certificates relation
unit-grafana-agent-66: 14:47:22 INFO juju.worker.uniter.operation ran "peers-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:22 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:22 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules: directory does not exist
unit-grafana-agent-66: 14:47:22 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:22 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f849415cdb0> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:22 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: Emitting Juju event logging_consumer_relation_created.
unit-grafana-agent-66: 14:47:22 INFO juju.worker.uniter.operation ran "logging-consumer-relation-created" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:22 INFO juju.worker.uniter found queued "leader-elected" hook
unit-grafana-agent-66: 14:47:23 DEBUG unit.grafana-agent/66.juju-log ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:23 DEBUG unit.grafana-agent/66.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules: directory does not exist
unit-grafana-agent-66: 14:47:23 DEBUG unit.grafana-agent/66.juju-log no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:23 DEBUG unit.grafana-agent/66.juju-log <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f3f267b4d60> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:23 DEBUG unit.grafana-agent/66.juju-log Emitting Juju event leader_elected.
unit-grafana-agent-66: 14:47:23 DEBUG unit.grafana-agent/66.juju-log Alert rules path does not exist: /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules
unit-grafana-agent-66: 14:47:23 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules: directory does not exist
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f5d769f4db0> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Emitting Juju event config_changed.
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Emitting custom event <COSAgentDataChanged via GrafanaAgentMachineCharm/COSAgentRequirer[cos-agent]/on/data_changed[37]>.
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/hwmon.rules
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/useful.rules
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/arp_cache.rules
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/memory.rules
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/network.rules
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/disk.rules
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/mdadm.rules
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/process.rules
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/pressure.rules
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/oomkill.rule
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/conntrack.rules
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/reboot.rules
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from src/prometheus_alert_rules/high_cpu_iowait.rule
unit-grafana-agent-66: 14:47:24 DEBUG unit.grafana-agent/66.juju-log updated alert rules file /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules/juju_gh-runner_eb824e40_grafana-agent.rules
unit-grafana-agent-66: 14:47:25 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules/juju_gh-runner_eb824e40_grafana-agent.rules
unit-grafana-agent-66: 14:47:25 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/loki_alert_rules/grafana_agent_high_rate.rule
unit-grafana-agent-66: 14:47:25 DEBUG unit.grafana-agent/66.juju-log Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/loki_alert_rules/high_error_rate.rule
unit-grafana-agent-66: 14:47:25 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:25 INFO juju.worker.uniter found queued "start" hook
unit-grafana-agent-66: 14:47:25 DEBUG unit.grafana-agent/66.juju-log ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:25 INFO unit.grafana-agent/66.juju-log Running legacy hooks/start.
unit-grafana-agent-66: 14:47:26 DEBUG unit.grafana-agent/66.juju-log ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:26 DEBUG unit.grafana-agent/66.juju-log Charm called itself via hooks/start.
unit-grafana-agent-66: 14:47:26 DEBUG unit.grafana-agent/66.juju-log Legacy hooks/start exited with status 0.
unit-grafana-agent-66: 14:47:26 DEBUG unit.grafana-agent/66.juju-log no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:26 DEBUG unit.grafana-agent/66.juju-log <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f94c1040e50> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:26 DEBUG unit.grafana-agent/66.juju-log Emitting Juju event start.
unit-grafana-agent-66: 14:47:26 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:27 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:27 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:27 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f74e1910e50> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:27 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: Emitting Juju event logging_consumer_relation_changed.
unit-grafana-agent-66: 14:47:27 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: Emitting custom event <LokiPushApiEndpointJoined via GrafanaAgentMachineCharm/LokiPushApiConsumer[logging-consumer]/on/loki_push_api_endpoint_joined[48]>.
unit-grafana-agent-66: 14:47:27 INFO juju.worker.uniter.operation ran "logging-consumer-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:27 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:27 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:27 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f4dd0eb0e00> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:27 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Emitting Juju event send_remote_write_relation_joined.
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Emitting custom event <PrometheusRemoteWriteEndpointsChangedEvent via GrafanaAgentMachineCharm/PrometheusRemoteWriteConsumer[send-remote-write]/on/endpoints_changed[54]>.
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/hwmon.rules
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/useful.rules
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/arp_cache.rules
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/memory.rules
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/network.rules
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/disk.rules
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/mdadm.rules
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/process.rules
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/pressure.rules
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/oomkill.rule
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/conntrack.rules
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/reboot.rules
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/high_cpu_iowait.rule
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: updated alert rules file /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules/juju_gh-runner_eb824e40_grafana-agent.rules
unit-grafana-agent-66: 14:47:28 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules/juju_gh-runner_eb824e40_grafana-agent.rules
unit-grafana-agent-66: 14:47:29 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules/juju_gh-runner_eb824e40_grafana-agent.rules
unit-grafana-agent-66: 14:47:29 INFO juju.worker.uniter.operation ran "send-remote-write-relation-joined" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:29 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:29 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:29 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f9ee3068e00> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:29 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Emitting Juju event send_remote_write_relation_changed.
unit-grafana-agent-66: 14:47:29 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Emitting custom event <PrometheusRemoteWriteEndpointsChangedEvent via GrafanaAgentMachineCharm/PrometheusRemoteWriteConsumer[send-remote-write]/on/endpoints_changed[60]>.
unit-grafana-agent-66: 14:47:29 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/hwmon.rules
unit-grafana-agent-66: 14:47:29 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/useful.rules
unit-grafana-agent-66: 14:47:29 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/arp_cache.rules
unit-grafana-agent-66: 14:47:29 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/memory.rules
unit-grafana-agent-66: 14:47:30 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/network.rules
unit-grafana-agent-66: 14:47:30 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/disk.rules
unit-grafana-agent-66: 14:47:30 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/mdadm.rules
unit-grafana-agent-66: 14:47:30 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/process.rules
unit-grafana-agent-66: 14:47:30 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/pressure.rules
unit-grafana-agent-66: 14:47:30 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/oomkill.rule
unit-grafana-agent-66: 14:47:30 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/conntrack.rules
unit-grafana-agent-66: 14:47:30 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/reboot.rules
unit-grafana-agent-66: 14:47:30 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from src/prometheus_alert_rules/high_cpu_iowait.rule
unit-grafana-agent-66: 14:47:30 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: updated alert rules file /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules/juju_gh-runner_eb824e40_grafana-agent.rules
unit-grafana-agent-66: 14:47:30 DEBUG unit.grafana-agent/66.juju-log send-remote-write:120: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules/juju_gh-runner_eb824e40_grafana-agent.rules
unit-grafana-agent-66: 14:47:30 INFO juju.worker.uniter.operation ran "send-remote-write-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:31 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:31 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:31 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f845ce64e50> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:31 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: Emitting Juju event grafana_dashboards_provider_relation_joined.
unit-grafana-agent-66: 14:47:31 INFO juju.worker.uniter.operation ran "grafana-dashboards-provider-relation-joined" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:31 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:31 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:31 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f4b82a84d60> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:31 DEBUG unit.grafana-agent/66.juju-log grafana-dashboards-provider:121: Emitting Juju event grafana_dashboards_provider_relation_changed.
unit-grafana-agent-66: 14:47:32 INFO juju.worker.uniter.operation ran "grafana-dashboards-provider-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:32 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:32 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:32 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7fc700a94e00> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:32 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: Emitting Juju event logging_consumer_relation_joined.
unit-grafana-agent-66: 14:47:32 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/loki_alert_rules/grafana_agent_high_rate.rule
unit-grafana-agent-66: 14:47:32 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/loki_alert_rules/high_error_rate.rule
unit-grafana-agent-66: 14:47:32 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: Emitting custom event <LokiPushApiEndpointJoined via GrafanaAgentMachineCharm/LokiPushApiConsumer[logging-consumer]/on/loki_push_api_endpoint_joined[76]>.
unit-grafana-agent-66: 14:47:33 INFO juju.worker.uniter.operation ran "logging-consumer-relation-joined" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:33 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:33 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:33 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7f7064cc4db0> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:33 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: Emitting Juju event logging_consumer_relation_changed.
unit-grafana-agent-66: 14:47:33 DEBUG unit.grafana-agent/66.juju-log logging-consumer:122: Emitting custom event <LokiPushApiEndpointJoined via GrafanaAgentMachineCharm/LokiPushApiConsumer[logging-consumer]/on/loki_push_api_endpoint_joined[82]>.
unit-grafana-agent-66: 14:47:33 INFO juju.worker.uniter.operation ran "logging-consumer-relation-changed" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7fccdfcd0e50> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Emitting Juju event cos_agent_relation_joined.
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Emitting custom event <COSAgentDataChanged via GrafanaAgentMachineCharm/COSAgentRequirer[cos-agent]/on/data_changed[88]>.
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/hwmon.rules
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/useful.rules
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/arp_cache.rules
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/memory.rules
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/network.rules
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/disk.rules
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/mdadm.rules
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/process.rules
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/pressure.rules
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/oomkill.rule
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/conntrack.rules
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/reboot.rules
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/high_cpu_iowait.rule
unit-grafana-agent-66: 14:47:34 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: updated alert rules file /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules/juju_gh-runner_eb824e40_grafana-agent.rules
unit-grafana-agent-66: 14:47:35 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules/juju_gh-runner_eb824e40_grafana-agent.rules
unit-grafana-agent-66: 14:47:35 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: updated alert rules file /var/lib/juju/agents/unit-grafana-agent-66/charm/loki_alert_rules/juju_gh-runner_eb824e40_github-runner.rules
unit-grafana-agent-66: 14:47:35 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/loki_alert_rules/juju_gh-runner_eb824e40_github-runner.rules
unit-grafana-agent-66: 14:47:35 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/loki_alert_rules/grafana_agent_high_rate.rule
unit-grafana-agent-66: 14:47:35 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/loki_alert_rules/high_error_rate.rule
unit-grafana-agent-66: 14:47:35 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: updated dashboard file /var/lib/juju/agents/unit-grafana-agent-66/charm/grafana_dashboards/juju_github_self-hosted_runner_metrics-cos-agent-github-runner-131.json
unit-grafana-agent-66: 14:47:35 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: updated dashboard file /var/lib/juju/agents/unit-grafana-agent-66/charm/grafana_dashboards/juju_github_self-hosted_runner_metrics_(long-term)-cos-agent-github-runner-131.json
unit-grafana-agent-66: 14:47:35 INFO juju.worker.uniter.operation ran "cos-agent-relation-joined" hook (via hook dispatching script: dispatch)
unit-grafana-agent-66: 14:47:35 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: ops 2.13.0 up and running.
unit-grafana-agent-66: 14:47:35 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: no relation on 'tracing': tracing not ready
unit-grafana-agent-66: 14:47:35 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: <class '__main__.GrafanaAgentMachineCharm'>.<property object at 0x7fe5b1284e00> returned None; quietly disabling charm_tracing for the run.
unit-grafana-agent-66: 14:47:35 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Emitting Juju event cos_agent_relation_changed.
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Emitting custom event <COSAgentDataChanged via GrafanaAgentMachineCharm/COSAgentRequirer[cos-agent]/on/data_changed[94]>.
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/hwmon.rules
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/useful.rules
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/arp_cache.rules
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/memory.rules
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/network.rules
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/disk.rules
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/mdadm.rules
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/process.rules
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/pressure.rules
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/oomkill.rule
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/conntrack.rules
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/reboot.rules
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from src/prometheus_alert_rules/high_cpu_iowait.rule
unit-grafana-agent-66: 14:47:36 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: updated alert rules file /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules/juju_gh-runner_eb824e40_grafana-agent.rules
unit-grafana-agent-66: 14:47:37 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/prometheus_alert_rules/juju_gh-runner_eb824e40_grafana-agent.rules
unit-grafana-agent-66: 14:47:37 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: updated alert rules file /var/lib/juju/agents/unit-grafana-agent-66/charm/loki_alert_rules/juju_gh-runner_eb824e40_github-runner.rules
unit-grafana-agent-66: 14:47:37 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/loki_alert_rules/juju_gh-runner_eb824e40_github-runner.rules
unit-grafana-agent-66: 14:47:37 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/loki_alert_rules/grafana_agent_high_rate.rule
unit-grafana-agent-66: 14:47:37 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: Reading alert rule from /var/lib/juju/agents/unit-grafana-agent-66/charm/loki_alert_rules/high_error_rate.rule
unit-grafana-agent-66: 14:47:37 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: updated dashboard file /var/lib/juju/agents/unit-grafana-agent-66/charm/grafana_dashboards/juju_github_self-hosted_runner_metrics-cos-agent-github-runner-131.json
unit-grafana-agent-66: 14:47:37 DEBUG unit.grafana-agent/66.juju-log cos-agent:131: updated dashboard file /var/lib/juju/agents/unit-grafana-agent-66/charm/grafana_dashboards/juju_github_self-hosted_runner_metrics_(long-term)-cos-agent-github-runner-131.json

Additional context

No response

[sed-i]

Thanks for reporting!

• There is some code duplication in Loki we haven't removed yet. Need to diff the logic to cos-lib, and perhaps update cos-lib to be applicable to loki as well.
• In charm code we are addressing validation but for some reason no debug log is printed.
• This could also be addressed by some sort of linting (github action; related). Could perhaps contribute to prometheus-community repo, inspired by:
  • https://github.com/FUSAKLA/promruval
  • https://github.com/cbrgm/promcheck
  • https://github.com/karancode/promtool-action