This is a continuation of my previous blog on Openstack and Docker. In this blog, I will cover Openstack Docker heat plugin and Magnum.

Following are some of the items that Nova Docker driver cannot do currently:

1. Passing environment variables
2. Linking containers
3. Specifying volumes
4. Orchestrating and scheduling the containers

Heat docker plugin solves problems 1-3 and partially solves problem 4. Following is the architecture diagram I found in Openstack Docker wiki for heat.

• Nova is not involved here. Openstack heat uses Docker plugin to talk to Docker agent on the host.
• The host here is the VM spawned. The VM can either be spawned by Nova or Heat can spawn this using Nova driver.
• Glance is not involved here as the container images are stored in Docker registry.
• The Heat approach allows us to specify environment variables, link containers, specify volumes as well as orchestrate the host on which the Docker runs.

Using Heat plugin:

I had some issues getting Heat plugin to work with Openstack Kilo. It worked fine in Openstack Icehouse, so I continued with that. I used the approach mentioned in this wiki to do the integration of Heat plugin with Openstack Icehouse in Devstack.

My environment:

Ubuntu 14.04 running on top of Virtualbox with Devstack Icehouse release. My localrc file is here. localrc has Nova network instead of Neutron.

Stacking and heat plugin install:

Do the stacking. After successful stacking, I followed the heat docker plugin installation as mentioned in the wiki. It is needed to restart heat engine service after the plugin installation. I normally do this by opening the screen session, goto the heat service screen session, kill the session(ctrl-c) and restart that particular service. After restarting heat engine, we can check if the plugin got installed and running.

$ heat resource-type-list | grep Docker
| DockerInc::Docker::Container            

Using heat to start local container:

Here, we will run docker agent on the host machine and then use the Openstack heat plugin to create containers. For this, the first step is to install docker on the host machine and allow http access for docker clients to connect. For docker installation, use the procedure here. By default, docker does not allow external http access. To start docker with external http access on port 2376, execute:

$ sudo /usr/bin/docker -d --host=tcp://0.0.0.0:2376

Following is a simple heat YML file to create nginx container on the localhost.

heat_template_version: 2013-05-23
description: >
  Heat template to deploy Docker containers to an existing host
resources:
  nginx-01:
    type: DockerInc::Docker::Container
    properties:
      image: nginx
      docker_endpoint: 'tcp://192.168.56.102:2376'

Lets create the heat stack using the above template file.

$ heat stack-create -f ~/heat/docker_temp.yml nginxheat1

Now check if the stack is successfully created:

$ heat stack-list
+--------------------------------------+---------------+-----------------+----------------------+
| id                                   | stack_name    | stack_status    | creation_time        |
+--------------------------------------+---------------+-----------------+----------------------+
| d878d8c1-ce17-4f29-9203-febd37bd8b7d | nginxheat1    | CREATE_COMPLETE | 2015-06-14T13:27:54Z |
+--------------------------------------+---------------+-----------------+----------------------

Check if container is running in the localhost:

$ docker -H :2376 ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
624ff5de9240        nginx:latest        "nginx -g 'daemon of   2 minutes ago       Up 2 minutes        80/tcp, 443/tcp     trusting_pasteur    

At this point, we can access the webserver running as container in localhost.

Using heat to start remote container:

Here, we will create a VM using Nova, start docker agent on the VM and then use Openstack heat plugin to create containers on the VM.

First, lets create a Fedora image and upload to Glance. I used the procedure here to create a Fedora image and upload to Glance. Lets look at glance images after this.

$ glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
| ID                                   | Name                            | Disk Format | Container Format | Size      | Status |
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+
| 17239070-5aef-4bab-85df-1f9f72b6370b | cirros-0.3.1-x86_64-uec         | ami         | ami              | 25165824  | active |
| ea6eb351-7268-4b2e-91cd-806a67c4e9fe | fedora-software-config          | qcow2       | bare             | 610140160 | active |
+--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+

Next, I used the following script to create Key needed for ssh and also to set appropriate security groups.

# Create key and upload
ssh-keygen
nova keypair-add --pub-key ~/.ssh/id_rsa.pub key1
# Permit ICMP (ping):
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
# Permit secure shell (SSH) access:
nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
# Permit 2375 port access (Docker endpoint):
nova secgroup-add-rule default tcp 2375 2375 0.0.0.0/0

Now, we can create Fedora VM using Nova.

nova boot --flavor m1.medium --image fedora-software-config --security-groups default --key-name key1 --max-count 1 fedoratest

It is necessary for the instance to have external internet access to download Docker images. With Nova networking, we need to tweak iptables to create nat filter for this. The first rule deletes default filter and second rule creates filter with eth0 address.

sudo iptables -t nat -D nova-network-snat -o br100 -s 10.0.0.0/24 -j SNAT --to-source 
sudo iptables -t nat -A nova-network-snat -o br100 -s 10.0.0.0/24 -j SNAT --to-source 

Lets check the instance created:

$ nova list
+--------------------------------------+------------+--------+------------+-------------+------------------+
| ID                                   | Name       | Status | Task State | Power State | Networks         |
+--------------------------------------+------------+--------+------------+-------------+------------------+
| ab701808-98fb-4cba-907f-663fd762cf2a | fedoratest | ACTIVE | -          | Running     | private=10.0.0.2 |
+--------------------------------------+------------+--------+------------+-------------+------------------+

To login to the fedora VM, we need to use the key which we used to create VM.

ssh -i ~/.ssh/id_rsa fedora@

Fedora image already comes with Docker agent installed. We need to start docker listening on 2375 port for Openstack heat to talk to it.

sudo cp /usr/lib/systemd/system/docker.service /etc/systemd/system/
Edit /etc/sysconfig/docker:
OPTIONS=--host=tcp://0.0.0.0:2375
sudo systemctl start docker.service

Now, we can create containers on the remote VM. Lets first specify the heat template to create nginx container on remote VM. The docker end is the remote VM ip address.

heat_template_version: 2013-05-23
description: >
  Heat template to deploy Docker containers to an existing host
resources:
  nginx-01:
    type: DockerInc::Docker::Container
    properties:
      image: nginx
      docker_endpoint: 'tcp://10.0.0.2:2375'

Lets start the heat stack using the above template.

$ heat stack-create -f docker_stack1.yml docker_stack2

It is needed to pull the nginx docker image on the VM before-hand. We can automate this step with heat.
Now, we can check that the container is created on the VM.

[fedora@fedoratest ~]$ docker -H :2375 ps
CONTAINER ID        IMAGE               COMMAND                CREATED              STATUS              PORTS               NAMES
18a7ed8f5b00        nginx:latest        "nginx -g 'daemon of   About a minute ago   Up About a minute   80/tcp, 443/tcp     tender_morse      

The heat template file mentioned here creates the VM, pulls apache, mysql containers and links them. Following is another good wiki which gets into the details of defining heat templates for docker containers and also gives a good example. I did not have luck getting these scripts working in my devstack environment. The instances got spawned, container creation failed. I spent some time, but could not figure it out.

Openstack Magnum:

Openstack heat plugin for Docker solves some of the problems with Nova Docker driver but it does not handle the dynamic Orchestration, scheduling. Magnum is a generic Container management solution being developed in Openstack to manage Docker as well as other Container technologies. Currently, Magnum supports Kubernetes and Docker. With both Kubernetes and Docker, cluster of hosts are created and Containers gets scheduled into the cluster using special clustering algorithms and constraints. Magnum is still at the early stages and the first release was done with Openstack Kilo few months back.

Following is Magnum architecture diagram mentioned in Magnum wiki.

• Magnum uses other Openstack components like Nova, Heat, Neutron, Glance, Keystone whereever needed to manage containers.
• Nova is used to create micro VM instances, containers run on top of it. Docker agent runs in each micro VM instance.
• Heat is used for overall orchestration. For Kubernetes case, Heat creates Kubernetes agents, replication controller. For Docker, Heat creates Docker swarm cluster, swarm agents.
• The steps involved are creation of bay-model(Kubernetes, Docker), bay creation and then container creation.

I tried to follow the procedure mentioned here to create Magnum cluster. I followed the steps till cluster creation, but cluster creation never completed successfully, it was stuck in “CREATE_IN_PROGRESS”. I tried with both Kubernetes and Docker baymodel, had no luck. If someone got this working, please let me know.

Container Orchestration is still evolving and there are multiple technologies being developed to address the problem. Docker is developing its own Orchestration solution. For folks who are already using Openstack, Magnum seems like a complete solution since it integrates well with other components of Openstack. I feel that over time, best technology would survive.

References:

• Openstack heat docker templates – 1, 2 and 3
• Openstack heat github page
• Magnum wiki
• Magnum github page
• Openstack Vancouver Magnum presentation

Note:

Pictures used in this blog are from references.