Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hook failed: "grafana-source-relation-departed" #281

Open
markbeierl opened this issue Dec 6, 2023 · 8 comments
Open

hook failed: "grafana-source-relation-departed" #281

markbeierl opened this issue Dec 6, 2023 · 8 comments
Labels
Status: Triage Type: Bug

Comments

@markbeierl
Copy link

Bug Description

Deployed cos lite bundle to a model and had it running for several days. Attempted to destroy the model, and grafana agent went into error status with hook failed: "grafana-source-relation-departed"

To Reproduce

  1. juju add-model cos microk8s
  2. juju deploy cos-lite --trust
  3. juju deploy cos-configuration-k8s
    --config git_repo=https://github.com/canonical/sdcore-cos-configuration
    --config git_branch=main
    --config git_depth=1
    --config grafana_dashboards_path=grafana_dashboards/sdcore/
  4. juju integrate cos-configuration-k8s grafana

I then performed cross model integrations with two other models on offer cos.prometheus:receive-remote-write and cos.loki:logging.

The first model, I performed a destroy model, however the second still remained with a relation.

I then destroyed the cos-lite model and this error occurred.

Environment

juju      3.1.6          24626  3.1/stable          canonical✓  -
microk8s  v1.27.7        6101   1.27-strict/stable  canonical✓  -

Relevant log output

unit-grafana-0: 16:59:34 INFO juju.worker.uniter awaiting error resolution for "relation-departed" hook
unit-grafana-0: 16:59:35 ERROR unit.grafana/0.juju-log grafana-source:13: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 1312, in <module>
    main(GrafanaCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 841, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 930, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_source.py", line 613, in _on_grafana_source_relation_departed
    removed_source = self._remove_source_from_datastore(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_source.py", line 628, in _remove_source_from_datastore
    stored_sources = self.get_peer_data("sources")
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_source.py", line 731, in get_peer_data
    data = self._charm.peers.data[self._charm.app].get(key, "")  # type: ignore[attr-defined]
AttributeError: 'NoneType' object has no attribute 'data'
unit-grafana-0: 16:59:35 ERROR juju.worker.uniter.operation hook "grafana-source-relation-departed" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

No response
@markbeierl
Copy link
Author

Additional info. Decided to see what happens on intervention.

Performed juju resolve grafana/0 --no-retry 3 times, checked the status between each and noticed that it changed to

grafana/0*  error     idle   10.1.212.218         hook failed: "ingress-relation-broken"

With this in debug-log

unit-grafana-0: 17:15:15 INFO juju.worker.uniter awaiting error resolution for "relation-broken" hook
unit-grafana-0: 17:15:16 ERROR unit.grafana/0.juju-log ingress:10: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 1312, in <module>
    main(GrafanaCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 841, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 930, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/traefik_route_k8s/v0/traefik_route.py", line 334, in _on_relation_broken
    self.on.ready.emit(event.relation)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 841, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 930, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 276, in _on_ingress_ready
    self._configure()
  File "./src/charm.py", line 436, in _configure
    if self._check_datasource_provisioning():
  File "./src/charm.py", line 403, in _check_datasource_provisioning
    grafana_datasources = self._generate_datasource_config()
  File "./src/charm.py", line 1023, in _generate_datasource_config
    for source_info in self.source_consumer.sources:
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_source.py", line 707, in sources
    stored_sources = self.get_peer_data("sources")
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_source.py", line 731, in get_peer_data
    data = self._charm.peers.data[self._charm.app].get(key, "")  # type: ignore[attr-defined]
AttributeError: 'NoneType' object has no attribute 'data'
unit-grafana-0: 17:15:16 ERROR juju.worker.uniter.operation hook "ingress-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1

@markbeierl
Copy link
Author

Next resolve --no-retry gives

unit-grafana-0: 17:16:34 ERROR unit.grafana/0.juju-log grafana-dashboard:23: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 1312, in <module>
    main(GrafanaCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 841, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 930, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1390, in _on_grafana_dashboard_relation_broken
    self._remove_all_dashboards_for_relation(event.relation)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1524, in _remove_all_dashboards_for_relation
    if self._get_stored_dashboards(relation.id):
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1557, in _get_stored_dashboards
    return self.get_peer_data("dashboards").get(str(relation_id), {})
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1572, in get_peer_data
    data = self._charm.peers.data[self._charm.app].get(key, "")  # type: ignore[attr-defined]
AttributeError: 'NoneType' object has no attribute 'data'
unit-grafana-0: 17:16:34 ERROR juju.worker.uniter.operation hook "grafana-dashboard-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1

@markbeierl
Copy link
Author

Two more resolves and

unit-grafana-0: 17:17:56 ERROR unit.grafana/0.juju-log metrics-endpoint:20: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 1312, in <module>
    main(GrafanaCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 841, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 930, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 370, in _maybe_provision_own_dashboard
    self.init_dashboard_provisioning(dashboards_dir_path)
  File "./src/charm.py", line 521, in init_dashboard_provisioning
    self._configure()
  File "./src/charm.py", line 436, in _configure
    if self._check_datasource_provisioning():
  File "./src/charm.py", line 403, in _check_datasource_provisioning
    grafana_datasources = self._generate_datasource_config()
  File "./src/charm.py", line 1023, in _generate_datasource_config
    for source_info in self.source_consumer.sources:
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_source.py", line 707, in sources
    stored_sources = self.get_peer_data("sources")
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_source.py", line 731, in get_peer_data
    data = self._charm.peers.data[self._charm.app].get(key, "")  # type: ignore[attr-defined]
AttributeError: 'NoneType' object has no attribute 'data'
unit-grafana-0: 17:17:56 ERROR juju.worker.uniter.operation hook "metrics-endpoint-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1

@PietroPasotti
Copy link
Contributor

Issue is that the peer relation is apparently removed before the grafana-source one. So when the charm lib does self._charm.peers, which does self.model.get_relation(PEER), it gets a None.
Solution: add guards in front of all self._charm.peers calls to early-exit if there is no peer relation (which hopefully only ever happens if the charm is being nuked)

@chanchiwai-ray chanchiwai-ray mentioned this issue Jan 25, 2024
Closed
@phvalguima
Copy link

Seeing this issue on grafana latest/stable, revision 113

@Abuelodelanada
Copy link
Contributor

In a chat with @gaborbk in Matrix we also see this issue on grafana latest/stable, revision 117

As far as I understand the issue is in fact in this method:

    def _get_stored_dashboards(self, relation_id: int) -> list:
        """Pull stored dashboards out of the peer data bucket."""
        return self.get_peer_data("dashboards").get(str(relation_id), {})

relation_id is None because we are calling this method from _remove_all_dashboards_for_relation. That method is called in the context of a relation-broken event where relation data can no longer be read-written.

unit-grafana-0: 16:03:38 DEBUG unit.grafana/0.juju-log grafana-dashboard:176: Emitting Juju event grafana_dashboard_relation_broken.
unit-grafana-0: 16:03:38 WARNING unit.grafana/0.juju-log grafana-dashboard:176: 'app' expected but not received.
unit-grafana-0: 16:03:38 WARNING unit.grafana/0.juju-log grafana-dashboard:176: 'app_name' expected in snapshot but not found.
unit-grafana-0: 16:03:38 ERROR unit.grafana/0.juju-log grafana-dashboard:176: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 1542, in <module>
    main(GrafanaCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 548, in main
    manager.run()
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 527, in run
    self._emit()
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 516, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/main.py", line 147, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 348, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 860, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-grafana-0/charm/venv/ops/framework.py", line 950, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 543, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1391, in _on_grafana_dashboard_relation_broken
    self._remove_all_dashboards_for_relation(event.relation)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 543, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1525, in _remove_all_dashboards_for_relation
    if self._get_stored_dashboards(relation.id):
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 543, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1558, in _get_stored_dashboards
    return self.get_peer_data("dashboards").get(str(relation_id), {})
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 543, in wrapped_function
    return callable(*args, **kwargs)  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/grafana_k8s/v0/grafana_dashboard.py", line 1573, in get_peer_data
    data = self._charm.peers.data[self._charm.app].get(key, "")  # type: ignore[attr-defined]
AttributeError: 'NoneType' object has no attribute 'data'
unit-grafana-0: 16:03:38 ERROR juju.worker.uniter.operation hook "grafana-dashboard-relation-broken" (via hook dispatching script: dispatch) failed: exit status 1

@Abuelodelanada
Copy link
Contributor

This issue is related to this fix in Juju 3.5.5:

https://bugs.launchpad.net/juju/+bug/1998282

@gaborbk
Copy link

gaborbk commented Dec 11, 2024

Thank you for finding the fix @Abuelodelanada - sounds like my only option is tiupgrade Juju version to 3.5.5+

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Status: Triage Type: Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Abuelodelanada @PietroPasotti @gaborbk @markbeierl @phvalguima