Deploying Python with Docker

5 min readFeb 1, 2015

Python applications can be easily and effectively deployed through Docker, but there are a few gotcha’s and best practices to make the process much more enjoyable. These are by no means the only way to do things, just the way that the team I work with has found to be the most usable and maintainable. Note: Most of the topics covered here are opinions. There are many different ways to do things in Docker (which is one of its benefits!). We don’t make frequent use of volume mounting which is a topic worth reading up on. Volume mounting can be used to dynamically mount the application source as a volume rather than having to run a build each time.

DEBIAN_FRONTEND

This environment variable should be well known to Docker users but for those who don’t, it signals to the operating system where to expect input from the user. By setting it to “noninteractive”, you can run commands without the process requesting input from the user. This is particularly useful when running apt-get commands which frequently prompt the user for various steps. Noninteractive mode will select the most appropriate default and allow for much quicker builds.

One gotcha is to make sure you only set this for specific RUN commands you are calling in a Dockerfile rather than using an ENV command to set it globally. ENV commands persist into the final running container which could cause problems when interacting with the container through BASH. Example:

# The Right Thing - This only sets the ENV var for this command
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y python3# The Wrong Thing - This sets the ENV var for everything below it, including the running container.
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get install -y python3

requirements.txt

Application dependancies change infrequently relative to the codebase, so installing these further up in the Dockerfile will speed up future builds where the only difference is a code change. By ensuring that the requirements.txt file is added and dependancies installed further up than the application source, Docker’s layered building of containers can easily cache the dependency installation step. This make each build phase significantly faster as dependancies do not have to be re-downloaded and re-built.

File Order

Building on the ideas above, it is important to look at the order in which files are added to the container. By putting more frequently edited files further down the folder, you can speed up the docker build phases by taking advantage of the container cacheing. For example, application configs, system configs, and dependancies all change pretty infrequently, so are prime candidates to move further up in the docker file. Source files such as routing files, views, and database code change more often and should therefore be be further down in the Dockerfile, especially below and Docker configuration commands such as EXPOSE, ENV, etc.

Putting some thought into how your files are copied into Docker won’t dramatically improve build speed, because most files aren’t used until the container is finished anyways (such as application source files), but it does prevent rebuilding stages of the container that are unchanged.

App Secrets

Safely and securely getting app secret keys to the application is something that we could not find a lot of information on before developing our system. What we ended up using was the env-file flag for the docker run command. We have a list of application secrets and configurations that are securely transferred to the deployment system in a file named app_config.list, they are then loaded into the docker container at runtime as follows

docker run -d -t -—env-file app_config.list <image:tag>

This method allows us to easily change application settings and secret keys without needing to rebuild a container.

NOTE: It is also important to ensure that app_config.list is in your .gitignore file so it does not get checked into source.

gunicorn

We use gunicorn as our application server inside containers. We’ve found it to be a stable and performant way to run our API’s. It provides a significant amount of configurability such as the ability to specify the number of workers and their types (green threads, gevent, etc). This would allow you to tune your application to it’s workload and get the best performance out of your container.

Setting up gunicorn is simple

# Installation
pip3 install gunicorn# Running the server
gunicorn api:app -w 4 -b 127.0.0.1:5000

Running the server as such is best for serving your site from behind NGINX with a reverse proxy. It also allows for easy load balancing.

supervisord

Do you need to have multiple processes running on your Docker container? Supervisord is going to be your goto tool. For examples sake, lets say we would like to deploy a container that serves a gunicorn application behind nginx. This could be done with some clever BASH scripting but lets do it a little more cleanly.

Supervisor is “a process control system that allows its users to monitor and control a number of processes on UNIX-like operating systems”. Sounds perfect! You’ll need to begin by installing supervisor in your docker container.

RUN DEBIAN_FRONTEND=noninteractive apt-get install -y supervisor

We’ll then need to write a config file for supervisor so it knows what to run and how to manage the processes.

[supervisord]
nodaemon = true  # This forces supervisor to run in the foreground[program:nginx]  # Name the first program we want to run
command = /usr/sbin/nginx  # Path to nginx executable
startsecs = 5  # We consider nginx to be successfully started if it stays up for 5 seconds[program:app-gunicorn]
command = gunicorn api -w 4 -b 127.0.0.1:5000
startsecs = 5

This is a very basic config and there are many configuration options available to control logging, stdout/stderr redirection, restart policies, etc. We’ve found it to be an invaluable tool in our deployments.

Once your config is written, make sure Docker copies it into the container.

ADD supervisord.conf /etc/supervisord.conf

And make supervisor the startup command for the container

CMD supervisord -c /etc/supervisord.conf

This will run both gunicorn and nginx when the container is started and, if configured, will restart them as needed.

Lessons Learned and Future Goals

We’ve spent a lot of time deploying code with Docker and I’m sure we’ll be spending a lot more. I’ll update this post as we learn more best-practices. One of the most important lessons we learned when working with Docker is to think minimally. It can be tempting to run your entire system inside of one container, but it is often more maintainable to run application processes in their own container. The most we usually have in a container is an application process and a webserver. In some cases, it is not beneficial to have nginx running inside the container, as there may be more complicated routing or configuration needed high up the stack, but for most cases, we find it to be an acceptable overhead inside the container.

I hope this information ends up being valuable in some way to others! As I mentioned, as we learn more as a team I’ll update this post to reflect the best-practices.