shell script to graceful restart gunicorn with source code reloading

I always thought restart gnicorn is as easy as kill -HUP <pid>, but there are two pitfalls:

First of all the code might be loaded in master process then fork'ed into workers, so the HUP signal won't reload the source code, only gunicorn config is updated.

Then there might be some bugs and your new code won't boot correctly, the worker will quit and drags down the master process along with it.

I discovered a new trick from github issues that can spawn new worker first, check if new code loads correctly then restart the whole rack.

According to the docs you need to send several signals and blah blah, the author tried to explain in great detail of how it works rather than providing a simple option to do so.

So I wrote a shell script to do that instead.

  #!/bin/bash

  oldpid=$(<./deploy/gunicorn.pid)
  # forget the old kill -HUP 
  kill -USR2 $oldpid
  sleep 1
  timeout=5
  while [[ $timeout -gt 0 ]]; do
    newpid=$(<./deploy/gunicorn.pid.2) && kill -0 $newpid
    if [[ $? -eq 0 ]]; then
      break
    else
      sleep 0.5
      (( timeout-- ))
    fi
  done;

  # shutdown old
  echo "success. newpid=$newpid"
  kill -WINCH $oldpid
  sleep 1
  kill -TERM $oldpid

The script will check for new master pid to be alive then kill the old master in a maximium timeout of 5 seconds.

It works perfectly to replace the simple HUP signal as intended.

Comments