shell script to graceful restart gunicorn with source code reloading
Posted | stdout
I always thought restart gnicorn is as easy as kill -HUP <pid>
, but there are two pitfalls:
First of all the code might be loaded in master process then fork'ed into workers, so the HUP
signal won't reload the source code, only gunicorn config is updated.
Then there might be some bugs and your new code won't boot correctly, the worker will quit and drags down the master process along with it.
I discovered a new trick from github issues that can spawn new worker first, check if new code loads correctly then restart the whole rack.
According to the docs you need to send several signals and blah blah, the author tried to explain in great detail of how it works rather than providing a simple option to do so.
So I wrote a shell script to do that instead.
#!/bin/bash
oldpid=$(<./deploy/gunicorn.pid)
# forget the old kill -HUP
kill -USR2 $oldpid
sleep 1
timeout=5
while [[ $timeout -gt 0 ]]; do
newpid=$(<./deploy/gunicorn.pid.2) && kill -0 $newpid
if [[ $? -eq 0 ]]; then
break
else
sleep 0.5
(( timeout-- ))
fi
done;
# shutdown old
echo "success. newpid=$newpid"
kill -WINCH $oldpid
sleep 1
kill -TERM $oldpid
The script will check for new master pid to be alive then kill the old master in a maximium timeout of 5 seconds.
It works perfectly to replace the simple HUP
signal as intended.
Comments