Skip to content

Commit

Permalink
fix: show degraded nodes in compromised nodes list in daily status re…
Browse files Browse the repository at this point in the history
…port
  • Loading branch information
louisevelayo committed Aug 29, 2024
1 parent 57cb216 commit e21891e
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 11 deletions.
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).


## [1.0.0-alpha.2] - 2023-12-12
## [1.0.0-alpha.2] - 2024-08-29

- Added logging.
- Fixed bug where `UNASSIGNED` nodes were being alerted on as being compromised.
Expand All @@ -20,6 +20,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Fixed an issue causing Node Monitor to go offline when a subscriber's email is not found.
- Added email subject line visibility in Slack and Telegram notifications.
- Fixed an issue where Telegram messages were not formatted properly.
- Fixed bug where `DEGRADED` nodes were not listed as compromised nodes in daily status report.
- Refactored messages to be more concise.


## [1.0.0-alpha.1] - 2023-10-20
Expand Down
18 changes: 8 additions & 10 deletions node_monitor/node_monitor_helpers/messages.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,31 +93,29 @@ def nodes_status_message(nodes: List[ic_api.Node],
nodes_up = [node for node in nodes if node.status == 'UP']
nodes_down = [node for node in nodes if node.status == 'DOWN']
nodes_unassigned = [node for node in nodes if node.status == 'UNASSIGNED']
nodes_disabled = [node for node in nodes if node.status == 'DISABLED']
nodes_degraded = [node for node in nodes if node.status == 'DEGRADED']
total_nodes = len(nodes)
nodes_compromised = [node for node in nodes
if node.status == 'DOWN' or node.status == 'DEGRADED']
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
def _make_diagnostic_message() -> str:
match len(nodes_down):
match len(nodes_compromised):
case 0: return ""
case _: return (f"Node(s) Compromised:\n"
f"\n"
f"{detailnodes(nodes_down, labels)}\n\n")
f"{detailnodes(nodes_compromised, labels)}\n\n")
def _make_subject() -> str:
datacenters = {node.dc_id.upper() for node in nodes_down}
match len(nodes_down):
datacenters = {node.dc_id.upper() for node in nodes_compromised}
match len(nodes_compromised):
case 0: return "🟢 All Systems Healthy"
case _: return "🟡 Action Required @ " + ', '.join(sorted(datacenters))
def _render_frac(numerator: int, denominator: int) -> str:
match numerator:
case 0: return "None"
case _: return f"{numerator}/{denominator}"
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
subject = _make_subject()
message = (
f"\nNode Provider: {nodes[0].node_provider_name}\n"
f"Nodes Total: {total_nodes}\n"
f"Node Health: {len(nodes_degraded) + len(nodes_down)} Unhealthy, {len(nodes_unassigned) + len(nodes_up)} Healthy\n"
f"Node Health: {len(nodes_degraded) + len(nodes_down)} Unhealthy, "
f"{len(nodes_unassigned) + len(nodes_up)} Healthy\n"
f"Node Assignment: {len(nodes_unassigned)} Unassigned, {len(nodes_up)} Assigned\n"
f"\n\n"
f"{_make_diagnostic_message()}"
Expand Down

0 comments on commit e21891e

Please sign in to comment.