Skip to content

Commit

Permalink
[designate] Remove RabbitMQ connection check from API health-probe
Browse files Browse the repository at this point in the history
Unfortunately, neither RabbitMQ connection is established
nor heartbeats are working before any request arrives
to the wsgi apache process.

So for this check to work some of the following is required:
- Constant stream of incoming requests triggering the connection to Rabbit
- API should run as a native eventlet application (designate-api)
- API should run as uwsgi application with pthreads enabled for heartbeats

Neither of these options is good for us, so disabling this part of health
probe for API service for now.
  • Loading branch information
s10 committed Nov 22, 2024
1 parent e5b41d1 commit 128317e
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 31 deletions.
2 changes: 1 addition & 1 deletion openstack/designate/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
apiVersion: v2
description: A Helm chart for Kubernetes to deploy Openstack Designate (DNSaaS)
name: designate
version: 0.4.0
version: 0.4.1
appVersion: "xena"
dependencies:
- condition: percona_cluster.enabled
Expand Down
31 changes: 1 addition & 30 deletions openstack/designate/bin/health-probe.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,7 @@
Uses oslo's ping method that is designed just for such simple purpose.
For API service script:
a. checks if there are any established RabbitMQ connections in a pod.
b. checks if the RabbitMQ and DB are reachable over TCP.
a. checks if the RabbitMQ and DB are reachable over TCP.
These two checks succeed or timeout within 5 seconds.
For other RPC services script:
Expand Down Expand Up @@ -138,21 +137,6 @@ def tcp_socket_status(process: Optional[str], ports: Set[int]) -> int:
return 0


def tcp_connection_established(ports: Set[int]) -> int:
"""Check the presence of established TCP connections to remote ports"""
scon = psutil.net_connections(kind="tcp")
for con in scon:
try:
rport = con.raddr[1]
status = con.status
except IndexError:
# Skip if raddr is empty tuple: connection is not ESTABLISED or connection is LISTEN
continue
if rport in ports and status == tcp_established:
return 1
return 0


def check_tcp_socket(
service: str,
rabbits: Set[Tuple[str, int]],
Expand Down Expand Up @@ -189,18 +173,6 @@ def check_tcp_socket(
sys.exit(1)


def check_rabbitmq_tcp_socket(rabbits: Set[Tuple[str, int]]) -> None:
"""Check RabbitMQ TCP socket in ESTABLISHED state is present"""
r_ports = {port for _, port in rabbits}
# Check that there are any established RabbitMQ connections
# Apache2 wsgi RabbitMQ connections doesn't belong to any process in pods netstat
if r_ports and tcp_connection_established(r_ports) == 0:
sys.stderr.write(f"RabbitMQ socket not established")
# Do not kill the pod if RabbitMQ is not reachable/down
if not cfg.CONF.liveness_probe:
sys.exit(1)


def check_tcp_connectivity(
rabbits: Set[Tuple[str, int]],
databases: Set[Tuple[str, int]],
Expand Down Expand Up @@ -292,7 +264,6 @@ def test_liveness() -> None:
check_service_status(transport)

if service in api_services:
check_rabbitmq_tcp_socket(rabbits)
check_tcp_connectivity(rabbits, databases)


Expand Down

0 comments on commit 128317e

Please sign in to comment.