The underlying infrastructure (1/3)

Each microservice is shipped in its own Docker container and we can deploy it on any machine which has installed the corresponding runtime. You can think of Docker containers like lightweight VMs that are based on arbitrary layers. You choose a base image, e.g. Ubuntu 14.04, and add software and files you need to run your application. Each command (copy files into it, install new packages, set an environment variable, ...) creates a new layer. The advantage is if I want to run my container on an other machine I only have to transfer the layers which are not already there. In the best case this is only the last layer containing your code changes.

Back to our overview, it is very nice to deploy your application in a container which requires only the Docker runtime because we don't have to care about the underlying hardware. To run a quick test, start the container on your local machine and for the deployment, do the same on the server without preparing the server by installing packages in a special version. The only thing you need is Docker.

Mesos 

Apache Mesos is an open-source project abstracting all hardware resources in your cluster.

Every slave tells the master how many CPUs, how much RAM and disk space it can offer.

The master collects all this information and acts as contact point for Mesos frameworks. At OTTO we use Marathon to run applications in the cluster. Mesos sends the collected hardware information to the framework and this selects a slave which offers the necessary hardware.

The Mesos master forwards the decision of the framework and the task (start Docker container with specified URL) to the slaves, which run the task and allocate the resources. Finally, the slaves send back their updated resources offer to the master.

It is important to know if you only want to run one instance of an application you can't distribute it over different slaves. You have to choose one which has the resources you need. But if you run your applications in HA mode with more than one instance you can tell Marathon to use a different slaves for every instance. If you don't set this option it could happen that all your instances are located on the same slave and when this one goes down your application is offline until Marathon restarts it on another slave.

One problem we encountered is that all applications running in the Mesos cluster can't persist any data. After a restart the application can't access the files it wrote to before. It is possible to mount a filesystem from the host but then you have to guarantee that your application is deployed to the same slave and that the same filesystem is assigned to it. To solve this problem we use databases running outside of the cluster in traditional VMs. This, however, is only a workaround for the next months.

Marathon

If you want to start an application in your Mesos cluster you can't send your requests directly to the master. You have to use a framework which knows the protocol to understand hardware offers from the slaves and to define your task. A very popular framework to do this is Marathon which provides a user friendly API to deploy Docker containers with their hardware requirements.

For every application you have to create a JSON configuration file which defines a unique id, required resources, the URL of the Docker container, etc. Marathon stores these configuration files and you can restart an application with an old one if you notice difficulties after deployment.

To start a new application or to restart a running application you have to use the Marthon REST API or the web UI. Both have the same functionality and so we use the former for automated processes, i.e. deployments in our shell scripts, and the latter for manual processes like scaling, restarting and troubleshooting.

If your application crashes, Marathon starts it again to avoid downtimes caused by manual processes. To use this feature every application has to implement a simple service which responses to health checks. This check can respond with an error code when internal processes go wrong or when the application should restart, e.g. to reload its configuration.

Another important feature is the rolling restart. You can configure how many instances should always be running which is necessary if you want to guarantee a zero downtime deployment.

Proxy

The third component in our infrastructure is a proxy server which also acts as a load balancer.

When Marathon starts or scales an application we create a new configuration with URLs depending on the application IDs. On the slaves every public port of an application is mapped to a random port to avoid clashes.

Deployment

Let's have a look at the deployment process to see how all components interact with each other.

1. Build a Docker container. First, you have to create a Dockerfile which describes your container: [code language="bash"]FROM ubuntu:14.04 RUN mkdir -p /var/opt/my-project ADD my-project.jar /var/opt/my-project/my-project.jar WORKDIR /var/opt/my-project CMD java -jar my-project.jar[/code] To build your container use the Docker cli: [code language="bash"]docker build -t my-project .[/code]
2. Store the container in your Dockerregistry. To let other people and Mesos use your container you have to publish it. Your company has to run an internal Docker registry for this purpose. [code language="bash"]docker push my-project[/code] 
3. Create a Marathon configuration file. A Marathon configuration file can have many parameters which describe how the application should be deployed, how many instances you need or which port your application should use. [code language="javascript"]{ "id": "/develop/my-project", "container": { "type": "DOCKER", ...} "cpus": 0.5, "mem": 512.0, "instances": 2, ... }[/code] To have a complete example look at the Marathon REST API documentation.
4. Deploy the application. The easiest way to deploy the application is to use the Marathon REST API. You can also try out the web UI but deployments should be an automated process which is why we prefer the former way. [code language="bash"]curl -s -X PUT -H "Content-Type: application/json" --connect-timeout 5 -d @config.json "${MARATHON_HOST}/v2/apps/${ID}"[/code]
   
5. Monitor the deployment If you want to know which state your deployment has you can use the Marathon web UI. Marathon will look for a slave that can satisfy your requirements. To finish the process the application has to reply to the health checks after the deployment. 
6. Update your DNS and proxy In most cases you want to connect your application outside of the Mesos cluster which is why you have to update the configuration of your DNS server and your proxy. Currently, we use a simple Jenkins pipeline which is triggered by Marathon.
7. Modify the configuration In the web UI you can scale up and down the number of instances, but to change any other parameter you have to use the REST API again. For example, if you want to deploy a new version of your project just create a new Docker container, change the URL in the Marathon configuration file and send this to Marathon. The instances with old configuration will shutdown after the new instances are deployed and healthy.