Redis connections reset by peer – container restart fixes?

[joestump]

Version of SearXNG, commit number if you are using on master branch and stipulate if you forked SearXNG

2024.11.10+dfaf5868e

How did you install SearXNG?

Official Docker container/instructions.

What happened?

Once every day or three SearXNG will start throwing Python exceptions saying its Redis connection is being reset by peer. This appears to be catastrophic in nature until the SearXNG container is restarted. NOTE: I do NOT restart Redis to remedy this issue – only the SearXNG container.

How To Reproduce

Unfortunately, I can't get this to reproduce with consistency. It does happen regularly now though – a few times a week – after running the container for a while (> 10 hours).

Expected behavior

Redis connections work correctly when Redis is up.

Screenshots & Logs

redis.exceptions.ConnectionError: Error while reading from int01.stump.wtf:6379 : (104, 'Connection reset by peer')
    raise ConnectionError(
  File "/usr/lib/python3.12/site-packages/redis/connection.py", line 520, in read_response
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
    response = connection.read_response()
  File "/usr/lib/python3.12/site-packages/redis/client.py", line 565, in parse_response
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return self.parse_response(conn, command_name, **options)
  File "/usr/lib/python3.12/site-packages/redis/client.py", line 525, in _send_command_parse_response
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    lambda: self._send_command_parse_response(
  File "/usr/lib/python3.12/site-packages/redis/client.py", line 549, in <lambda>
           ^^^^
    return do()
  File "/usr/lib/python3.12/site-packages/redis/retry.py", line 62, in call_with_retry
    raise error
  File "/usr/lib/python3.12/site-packages/redis/client.py", line 538, in _disconnect_raise
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    lambda error: self._disconnect_raise(conn, error),
  File "/usr/lib/python3.12/site-packages/redis/client.py", line 552, in <lambda>
    fail(error)
  File "/usr/lib/python3.12/site-packages/redis/retry.py", line 65, in call_with_retry
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return conn.retry.call_with_retry(
  File "/usr/lib/python3.12/site-packages/redis/client.py", line 548, in execute_command
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return self.execute_command("GET", name)
  File "/usr/lib/python3.12/site-packages/redis/commands/core.py", line 1821, in get
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    token = redis_client.get(TOKEN_KEY)
  File "/usr/local/searxng/searx/botdetection/link_token.py", line 148, in get_token
                           ^^^^^^^^^^^^^^^^^^^^^^
    kwargs['link_token'] = link_token.get_token()
  File "/usr/local/searxng/searx/webapp.py", line 388, in render
           ^^^^^^^
    return render(
  File "/usr/local/searxng/searx/webapp.py", line 606, in index
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 865, in dispatch_request
         ^^^^^^^^^^^^^^^^^^^^^^^
    rv = self.dispatch_request()
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 880, in full_dispatch_request
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 882, in full_dispatch_request
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    response = self.full_dispatch_request()
  File "/usr/lib/python3.12/site-packages/flask/app.py", line 1473, in wsgi_app
Traceback (most recent call last):
During handling of the above exception, another exception occurred:
ConnectionResetError: [Errno 104] Connection reset by peer
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    data = self._sock.recv(socket_read_size)
  File "/usr/lib/python3.12/site-packages/redis/_parsers/socket.py", line 65, in _read_from_socket
    self._read_from_socket()

Additional context

I've set limiter to false, which I saw referenced in an older ticket as a possible remedy. What I find odd is that the get_token function seems to short-circuit if it can't connect. I have limiter to false, so I'm unsure why it's getting to line 148. I'm wondering if it would make more sense to base the get_token logic on the limiter value in the config or the value of SEARXNG_REDIS_URL being None?

[return42]

I have limiter to false, so I'm unsure why it's getting to line 148.

May option public_instance is activated. BTW redis is not only used by the limiter .. its not recommended to run a SearXNG instance without a Key/Value database (redis or valkey).

---

NOTE: I do NOT restart Redis to remedy this issue – only the SearXNG container.

I'm not sure if we should move the reported error to searxng-docker ... I had not observed that SearXNG loses the connection so far ... maybe it is due to the containers?

BTW the official SearXNG docker imag uses valkey --> may your issue is related to?