Netbox Cache Sentinel Read Timeout Exception when trying get app config #16784
Unanswered
phandc
asked this question in
Help Wanted!
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hi all,
Recently, I have deployed 3 Netbox applications to production. To maintain HA criteria, I also set up Redis Sentinel alongside Redis.
I observe strange behaviors from the log. The first time I started Netbox containers, it worked perfectly as expected.
After an amount of time a new request comes to the applications, it raises the "ConnectionInterrupted
Redis TimeoutError: Timeout reading from socket".
Following the traceback, I discovered that Netbox has a middleware called "MaintenanceModeMiddleware".
Every time, a request comes the middleware also runs and it calls "get config()" to get the current app config.
The app config is obtained from the "Redis cache config key" or the Postgres database.
The code below demonstrates this idea:
As you can see, a middleware will get the config value from the Redis cache every time the system receives a request.
I don't why after amount of time, the connection was broken and it raised the timeout exception.
I tried to add socket_connect_timeout and socket_timeout to avoid the Django Server waiting infinitely.
That will cause the 504 error in the client side.
Here is my redis cache sentinel config.
Although I had set socket_timeout to 5 seconds, the Redis cache read timeout still happens. I don't know the reason why.
Beta Was this translation helpful? Give feedback.
All reactions