uwsgi.reload() is causing downtime when it reloads

[AshutoshS777]

Problem Statement: uwsgi.reload() is causing downtime when it reloads.

Expected Outcome: Reload worker threads without any down time. In essence, system should stagger killing and reinstating worker threads one at a time, ensuring no incoming requests are missed/failed.

Can someone advise if reload is the right method achieving this outcome, if not, what is the ideal solution ??

Logs after calling uwsgi.reload() method:-

HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 1)
...gracefully killing workers...
Gracefully killing worker 2 (pid: 50529)...
Gracefully killing worker 4 (pid: 50541)...
Gracefully killing worker 3 (pid: 50535)...
Gracefully killing worker 5 (pid: 50547)...
Gracefully killing worker 6 (pid: 50553)...
Gracefully killing worker 10 (pid: 50577)...
Gracefully killing worker 9 (pid: 50571)...
Gracefully killing worker 1 (pid: 50523)...
Gracefully killing worker 7 (pid: 50559)...
Gracefully killing worker 8 (pid: 50562)...
[pid: 50562|app: 0|req: 383/3405] 10.128.2.1 () {60 vars in 1040 bytes} [Fri Dec 21 10:01:19 2018] GET /APIURL/DUMMAYNAME => generated 69 bytes in 5841 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 1)
worker 8 buried after 1 seconds
Fri Dec 21 10:02:26 2018 - worker 1 (pid: 50523) is taking too much time to die...NO MERCY !!!
Fri Dec 21 10:02:26 2018 - worker 2 (pid: 50529) is taking too much time to die...NO MERCY !!!
Fri Dec 21 10:02:26 2018 - worker 3 (pid: 50535) is taking too much time to die...NO MERCY !!!
Fri Dec 21 10:02:26 2018 - worker 4 (pid: 50541) is taking too much time to die...NO MERCY !!!
Fri Dec 21 10:02:26 2018 - worker 5 (pid: 50547) is taking too much time to die...NO MERCY !!!
Fri Dec 21 10:02:26 2018 - worker 6 (pid: 50553) is taking too much time to die...NO MERCY !!!
Fri Dec 21 10:02:26 2018 - worker 7 (pid: 50559) is taking too much time to die...NO MERCY !!!
Fri Dec 21 10:02:26 2018 - worker 9 (pid: 50571) is taking too much time to die...NO MERCY !!!
Fri Dec 21 10:02:26 2018 - worker 10 (pid: 50577) is taking too much time to die...NO MERCY !!!

Then it again initialized all worker together

** Starting uWSGI 2.0.17.1-debian (64bit) on [Thu Dec 20 10:02:07 2018] ***
compiled with version: 8.2.0 on 02 October 2018 08:22:00
os: Linux-3.10.0-862.11.6.el7.x86_64 #1 SMP Fri Aug 10 16:55:11 UTC 2018
nodename: nodename
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 16
current working directory: /home/uwsgi
detected binary path: /usr/bin/uwsgi-core
chdir() to /usr/local/share/
your processes number limit is 1048576
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited INET address :8000 fd 6
Python version: 3.6.7rc1 (default, Sep 27 2018, 09:51:25) [GCC 8.2.0]
Python main interpreter initialized at 0x5774850d6530
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 917136 bytes (895 KB) for 20 cores
*** Operational MODE: preforking+threaded ***
Using TensorFlow backend.
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x5654850d6530 pid: 7 (default app)
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 7)
spawned uWSGI worker 1 (pid: 50523, cores: 2)
mapping worker 1 to CPUs: 0 1 2 3
spawned uWSGI worker 2 (pid: 50529, cores: 2)
mapping worker 2 to CPUs: 4 5 6 7
spawned uWSGI worker 3 (pid: 50535, cores: 2)
mapping worker 3 to CPUs: 8 9 10 11
spawned uWSGI worker 4 (pid: 50541, cores: 2)
mapping worker 4 to CPUs: 12 13 14 15
spawned uWSGI worker 5 (pid: 50547, cores: 2)
mapping worker 5 to CPUs: 0 1 2 3
mapping worker 6 to CPUs: 4 5 6 7
spawned uWSGI worker 6 (pid: 50553, cores: 2)
spawned uWSGI worker 7 (pid: 50559, cores: 2)
mapping worker 7 to CPUs: 8 9 10 11
spawned uWSGI worker 8 (pid: 50562, cores: 2)
mapping worker 8 to CPUs: 12 13 14 15
spawned uWSGI worker 9 (pid: 50571, cores: 2)
mapping worker 9 to CPUs: 0 1 2 3
spawned uWSGI worker 10 (pid: 50577, cores: 2)
mapping worker 10 to CPUs: 4 5 6 7
subprocess 50479 exited with code 0

[akx]

There's an article about the various ways you can reload your app with:

https://uwsgi-docs.readthedocs.io/en/latest/articles/TheArtOfGracefulReloading.html

[AshutoshS777]

@akx we implemented uwsgi.reload() for "graceful reloading”.
we expected it will kill one worker thread and reinitialize then it goes to other thread so that there wont be any downtime but in my case it is killing all worker threads and initializing all worker threads thus resulting in downtime for 30 seconds.