Health points returns OK when Synapse is integrated with PostgreSQL but the database is unreachable

[amandahla]

Description

While testing the Synapse behavior during a database downtime, I noticed that the health endpoint still returns OK even if Synapse can't handle events and there are errors in the log such as "psycopg2.OperationalError: connection to server at "11.1.1.1", port 5432 failed: Connection refused"

Steps to reproduce

• Install Synapse
• Configure to use a PostgreSQL database
• Stop the database
• Reach the /health endpoint

Homeserver

Local

Synapse Version

1.101.0

Installation Method

Other (please mention below)

Database

PostgreSQL single server

Workers

Single process

Platform

Microk8s (kubernetes cluster)

Configuration

Default configurations

database:
  args:
    cp_max: 10
    cp_min: 5
    database: synapse
    host: 11.1.1.1
    password: password
    port: '5432'
    user: relation-3
  name: psycopg2

Relevant log output

2024-07-31T14:11:18.545Z [synapse] Traceback (most recent call last):
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/synapse/metrics/background_process_metrics.py", line 251, in run
2024-07-31T14:11:18.545Z [synapse]     return await func(*args, **kwargs)
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/synapse/storage/databases/main/client_ips.py", line 492, in _prune_old_user_ips
2024-07-31T14:11:18.545Z [synapse]     await self.db_pool.runInteraction(
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/synapse/storage/database.py", line 951, in runInteraction
2024-07-31T14:11:18.545Z [synapse]     return await delay_cancellation(_runInteraction())
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks
2024-07-31T14:11:18.545Z [synapse]     result = context.run(
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator
2024-07-31T14:11:18.545Z [synapse]     return g.throw(self.value.with_traceback(self.tb))
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/synapse/storage/database.py", line 917, in _runInteraction
2024-07-31T14:11:18.545Z [synapse]     result = await self.runWithConnection(
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/synapse/storage/database.py", line 1046, in runWithConnection
2024-07-31T14:11:18.545Z [synapse]     return await make_deferred_yieldable(
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/twisted/python/threadpool.py", line 269, in inContext
2024-07-31T14:11:18.545Z [synapse]     result = inContext.theWork()  # type: ignore[attr-defined]
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/twisted/python/threadpool.py", line 285, in <lambda>
2024-07-31T14:11:18.545Z [synapse]     inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/twisted/python/context.py", line 117, in callWithContext
2024-07-31T14:11:18.545Z [synapse]     return self.currentContext().callWithContext(ctx, func, *args, **kw)
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/twisted/python/context.py", line 82, in callWithContext
2024-07-31T14:11:18.545Z [synapse]     return func(*args, **kw)
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/twisted/enterprise/adbapi.py", line 280, in _runWithConnection
2024-07-31T14:11:18.545Z [synapse]     conn = self.connectionFactory(self)
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/twisted/enterprise/adbapi.py", line 34, in __init__
2024-07-31T14:11:18.545Z [synapse]     self.reconnect()
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/twisted/enterprise/adbapi.py", line 71, in reconnect
2024-07-31T14:11:18.545Z [synapse]     self._connection = self._pool.connect()
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/twisted/enterprise/adbapi.py", line 410, in connect
2024-07-31T14:11:18.545Z [synapse]     conn = self.dbapi.connect(*self.connargs, **self.connkw)
2024-07-31T14:11:18.545Z [synapse]   File "/usr/local/lib/python3.10/dist-packages/psycopg2/__init__.py", line 122, in connect
2024-07-31T14:11:18.545Z [synapse]     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
2024-07-31T14:11:18.545Z [synapse] psycopg2.OperationalError: connection to server at "11.1.1.1", port 5432 failed: Connection refused
2024-07-31T14:11:18.545Z [synapse] 	Is the server running on that host and accepting TCP/IP connections?

### Anything else that would be useful to know?

_No response_

[dklimpel]

Duplicate to:

• Make healthcheck check a little bit more #11473

[devonh]

Thank you for taking the time to raise this.
It is indeed a duplicate of #11473 as mentioned so I will close this issue.