Category Archives: Continuous Delivery

Developing a Modern Distributed System – Part II: Provisioning with Docker

As described in an earlier blog post “Bootstrapping the Project”, comSysto’s performance and continuous delivery guild members are currently evaluating several aspects of distributed architectures in their spare time left besides customer project work. In the initial lab, a few colleagues of mine built the starting point for a project called Hash-Collision:

Hash-collision's system structure

They focused on the basic application structure and architecture to get a showcase up and running as quickly as possible and left us the following tools for running it:

  • one simple shell script to set up the environment locally based on many assumptions
  • one complex shell script that builds and runs all services
  • hardcoded dependency on a local RabbitMQ installation

First Attempt: Docker containers as a runtime environment

In search of a more sophisticated runtime environment we went down the most obvious path and chose to get our hands on the hype technology of late 2014: Docker. I assume that most people have a basic understanding of Docker and what it is, so I will not spend too much time on its motivation here. Basically, it is a tool inspired by the idea of ‘write once, run anywhere’, but on a higher level of abstraction than that famous programming language. Docker can not only make an application portable, it allows to ship all dependencies such as web servers, databases and even operating systems as one or multiple well-defined images and use the very same configuration from development all the way to production. Even though we did not even have any production or pre-production environments, we wanted to give it a try. Being totally enthusiastic about containers, we chose the most container-like place we could find and locked ourselves in there for 2 days.


One of the nice things about Docker is that it encourages developers to re-use existing infrastructure components by design. Images are defined incrementally by selecting a base image, and building additional functionality on top of it. For instance, the natural way to create a Tomcat image would be to choose a base image that already brings a JDK and install Tomcat on top of it. Or even simpler, choose an existing Tomcat image from the Docker Hub. As our services are already built as fat JARs with embedded web servers, things were almost trivial.

Each service should run in a standardized container with the executed JAR file being the sole difference. Therefore, we chose to use only one service image and inject the correct JAR using Docker volumes for development. On top of that, we needed additional standard containers for nginx (dockerfile/nginx) and RabbitMQ (dockerfile/rabbitmq). Each service container has a dependency on RabbitMQ to enable communication, and the nginx container needs to know where the Routing service resides to fulfill its role as a reverse proxy. All other dependencies can be resolved at runtime via any service discovery mechanism.

As a first concrete example, this is the Dockerfile for our service image. Based on Oracle’s JDK 8, there is not much left to do except for running the JAR and passing in a few program arguments:

FROM dockerfile/java:oracle-java8
MAINTAINER Christian Kroemer (
CMD /bin/sh -c 'java -Dport=${PORT} -Damq_host=${AMQ_PORT_5672_TCP_ADDR} -Damq_port=${AMQ_PORT_5672_TCP_PORT} -jar /var/app/app.jar'

After building this image, it is ready for usage in the local Docker repository and can be used like this to run a container:

# start a new container based on our docker-service-image
docker run docker-service-image
# link it with a running rabbitmq container to resolve the amq dependency
docker run --link rabbitmq:amq docker-service-image
# do not block and run it in background
docker run --link rabbitmq:amq -d docker-service-image
# map the container http port 7000 to the host port 8000
docker run --link rabbitmq:amq -d -p 8000:7000 docker-service-image
# give an environment parameter to let the embedded server know it has to start on port 7000
docker run --link rabbitmq:amq -d -e "PORT=7000" -p 8000:7000 docker-service-image
# inject the user service fat jar
docker run --link rabbitmq:amq -d -e "PORT=7000" -v HASH_COLLISION_PATH/user/build/libs/user-1.0-all.jar:/var/app/app.jar -p 8000:7000 docker-service-image

Very soon we ended up with a handful of such bash commands we pasted into our shells over and over again. Obviously we were not exactly happy with that approach and started to look for more powerful tools in the Docker ecosystem and stumbled over fig (which was not yet deprecated in favor of docker-compose at that time).

Moving on: Docker Compose for some degree of service orchestration

Docker-compose is a tool that simplifies the orchestration of Docker containers all running on the same host system based on a single docker installation. Any `docker run` command can be described in a structured `docker-compose.yml` file and a simple `docker-compose up` / `docker-compose kill` is enough to start and stop the entire distributed application. Furthermore, commands such as `docker-compose logs` make it easy to aggregate information for all running containers.


Here is an excerpt from our `docker-compose.yml` that illustrates how self-explanatory those files can be:

 image: dockerfile/rabbitmq
 - ":5672"
 - "15672:15672"
 build: ./service-image
 - "8000:7000"
 - ../user/build/libs/user-1.0-all.jar:/var/app/app.jar
 - PORT=7000
 - rabbitmq:amq

Semantically, the definition of the user service is equivalent to the last sample command given above except for the handling of the underlying image. The value given for the `build` key is the path to a directory that contains a `Dockerfile` which describes the image to be used. The AMQ service, on the other hand, uses a public image from the Docker Hub and hence uses the key `image`. In both cases, docker-compose will automatically make sure the required image is ready to use in the local repository before starting the container. A single `docker-compose.yml` file consisting of one such entry for each service is now sufficient for starting up the entire distributed application.

An Aside: Debugging the application within a Docker container

For being able to debug an application running in a Docker container from the IDE, we need to take advantage of remote debugging as for any physical remote server. For doing that, we defined a second service debug image with the following `Dockerfile`:

FROM dockerfile/java:oracle-java8
MAINTAINER Christian Kroemer (
CMD /bin/sh -c 'java -Xdebug -Xrunjdwp:transport=dt_socket,address=10000,server=y,suspend=n -Dport=${PORT} -Damq_host=${AMQ_PORT_5672_TCP_ADDR} -Damq_port=${AMQ_PORT_5672_TCP_PORT} -jar /var/app/app.jar'

This will make the JVM listen for a remote debugger on port 10000 which can be mapped to any desired host port as shown above.

What we got so far

With a local installation of Docker (on a Mac using boot2docker and docker-compose, starting up the whole application after checking out the sources and building all JARs is now as easy as:

  • boot2docker start (follow instructions)
  • docker-compose up -d (this will also fetch / build all required images)
  • open http://$(boot2docker ip):8080/overview.html

Note that several problems originate from boot2docker on Mac. For example, containers can not be accessed on `localhost`, but using the IP of a VM as boot2docker is built using a VirtualBox image.

In an upcoming blog post, I will outline one approach to migrate this setup to a more realistic environment with multiple hosts using Vagrant and Ansible on top. Until then, do not forget to check if Axel Fontaine’s Continuous Delivery Training at comSysto is just what you need to push your knowledge about modern deployment infrastructure to the next level.


Developing a Modern Distributed System – Part III: Provisioning with Docker, Vagrant and Ansible

Part II of our blog post series on ‘Developing a Modern Distributed System’ featured our first steps with Docker. In a second lab in early 2015, we tried to better understand the required changes in a production-like deployment. Without the assumption of all containers running on the same host – which makes no sense for a scalable architecture – Docker links and docker-compose are no longer valid approaches. We wanted to get the following three-node setup to work:


First of all, we created an automated Docker-Hub build linked to our github repository for rebuilding images on each commit. With that, the machines running the containers no longer had to build the images from Dockerfiles themselves. We used Vagrant to run three standard Ubuntu VMs and Ansible to provision them which included:

  • install Docker
  • upload service JARs that should be linked into the containers
  • upload static resources for nginx’s `/var/www` folder
  • run docker containers with correct parameterization (as we did with docker-compose before), still with some hardcoding to wire up different hosts

Why Ansible? First, some tool is required to avoid manually typing commands via ssh on multiple simultaneous sessions in an environment with multiple hosts. Second, Ansible was an easy choice because some of us already had experience using it while others wanted to give it a try. And last but not least, labs at comSysto are just the right place to experiment with unconventional combinations, see where their limitations are and prove they can still work! We actually achieved that, but after a `vagrant destroy` it took a full 20min on a developer machine to be up and running again. Not exactly the round-trip time you want to have while crafting your code… We needed to optimize.

The biggest improvement we were able to achieve came from the usage of a custom Vagrant base box with a ready-to-go Docker environment. Besides installing docker, we also pre-fetched all images from the Docker Hub right away which brings a huge productivity boost on slow internet connections. Even if images change, the large base images are typically pretty stable, hence download times could be reduced dramatically. The Docker image itself could also be optimized by using a minimal JDK base image such as jeanblanchard/busybox-java:8 instead of dockerfile/java:oracle-java8 which is built on top of Ubuntu.

Furthermore, we used CoreOS instead of Ubuntu as the operating system to get the base box smaller and faster to start up. CoreOS is a minimal OS designed to run Docker containers and do pretty much nothing on top of that. That also means it does not contain Python which is required to provision the VM using Ansible. Fortunately, Ansible can be installed using a specific coreos-bootstrap role.

Provisioning the running VMs with updated versions of our services, instead of destroying and rebuilding them from scratch, gave us a round-trip-time of roughly more than a minute, of which around 30s were required to rebuild all fat JARs.

Let’s have a closer look at a few aspects of that solution. First, we start a standard CoreOS box with Vagrant, and provision it with the following Ansible playbook:

- hosts: all
 gather_facts: False
 - defunctzombie.coreos-bootstrap
- hosts: all
 gather_facts: False
 - name: Prepare latest Docker images to make start of fresh VMs fast in case these are still up-to-date
 command: docker pull {{item}}
 - dockerfile/rabbitmq
 - chkcomsysto/hash-collision-service
 - chkcomsysto/hash-collision-service-debug
 - chkcomsysto/hash-collision-nginx

Using `vagrant package` and `vagrant box add` we immediately create a snapshot of that VM state and make it available as a base box for further usage. After that, the VM has fulfilled its destiny and is destroyed right away. The new base box is used in the `Vagrantfile` of the environment for our application which, again, is provisioned using Ansible. Here is a snippet of the corresponding playbook:

- hosts: service
 sudo: yes
 gather_facts: False
 - name: Create directory for runnable JARs
 path: /var/hash-collision
 state: directory
 - name: Upload User service runnable JAR
 src: ../../../user/build/libs/user-1.0-all.jar
 dest: /var/hash-collision/user-1.0-all.jar
 - name: Pull latest Docker images for all services
 command: docker pull chkcomsysto/hash-collision-service-debug
 - name: Start User service as a Docker container
 command: >
 docker run -d
 -p 7000:7000 -p 17000:10000
 --expose 7000 --expose 10000
 -e PORT=7000 -e HOST= -e AMQ_PORT_5672_TCP_ADDR= -e AMQ_PORT_5672_TCP_PORT=5672
 -v /var/hash-collision/user-1.0-all.jar:/var/app/app.jar

Where this leaves us

As we have virtualized pretty much everything, the only prerequisite left was a local Vagrant installation based on VirtualBox. After running a `` script to build the Vagrant base box from scratch once, executing a `` script was sufficient to build the application, start up three VMs with Vagrant, provision them with Ansible, and insert sample data. For a full development round-trip on a running system, another `` script was meant to build the application, provision the running VMs with Ansible, and again insert sample data (not that this is always required as we were still using in-memory storage without any persistence).

This setup allows us to run the entire distributed application in a multi-host environment during development. A more realistic approach would of course be to work on one component at a time, verify its implementation via unit tests and integrate it by starting this particular service from the IDE configured against an environment that contains the rest of the application. It is easy to see how our solution could be modified to support that use case.

Next Steps & Challenges

For several reasons, the current state feels pretty immature. First, we need to inject the own IP and global port into each container. It is questionable if a container should even need to know its identity. An alternative would be to get rid of the heartbeat-approach in which every service registers itself and build a service discovery based on Docker meta data with etcd or the likes instead.

Another area for improvements is our almost non-existent delivery pipeline. While uploading JARs into VMs is suitable for development, it is far from ideal for a production delivery. The Docker image should be self-contained, but this requires a proper build pipeline and artifact repository that automates all the way from changes in the service code to built JARs and fully functional Docker images ready to be started anywhere. Non-local deployments, e.g. on AWS, are also an interesting area of research in which the benefits of Docker are supposed to shine.

Last but not least, we need to work on all kinds of monitoring which is a critical part of any distributed application. Instead of using ssh to connect to a VM or remote server and then ssh again into the containers running there to see anything, it would be more appropriate to include a dedicated log management service (e.g. ELK) and send all logs there right away. On top of that, well-defined metrics to monitor the general health and state of services can be an additional source of information.

Obviously, there is a lot left to explore and learn in upcoming labs! Also crazy about DevOps, automation and self-contained deployment artifacts instead of 20th-century-style-delivery? See it all in action at our Continuous Delivery Training!

Introduction to comSysto´s cS ONE Conference

We were planning this amazing event for a couple month. On December 4th, 2014 it was finally time to start with the cS One Conference. This event was the first ever internal conference for comSysto, therefore everyone was enthusiastic and very excited about the outcome.

The Idea Behind It / Motivation

The introduction to the conference was made during breakfast by Daniel Bartl, one of the owners of comSysto.

Featured image      Featured image

As a proponent of the idea “New Work, New Culture”, we always try to find a way how to give our colleagues a chance to do something they deeply care about and love, to transfer knowledge to each other, to socialize with each other and be able to work in teams together.

The cS ONE Conference was all about knowledge transfer and team strengthening. comSysto employees had the chance to organize their own workshop or bring certain topic up for discussion in a group setting, which was conducted during working hours. The employees were very passionate about their workshops and group discussions and were looking forward to the conference. Everyone had the chance to sign up for the workshops and talks which, took place that day.

The Agenda

In order to start our day with lots of energy, we kicked it off with great breakfast which was very delicious and kept us going until lunch break thanks to lunchbox catering.

Featured image      Featured image

The talks as well as the workshops started both at 10 am. See below the timesheet for each talk and workshop.

Featured image      Featured imageThe topics of the talks were as follows:

  • JVM Deep Dive
  • Angular JS
  • Footfalls reloaded (Talk from Strata Barcelona)
  • Stress management & Burnout prevention
  • Agile at Emnos
  • DIY IOT Sensor Labs

The topics of the workshops were as follows:

  • How groups of people work
  • Shortcut Mania
  • comSysto moodboard
  • Modern Dev Tools
  • Building a mobile web app with
  • MOOC Mania

The topics of the tables were as follows:

  • comSysto Continuous Improvement Board
  • Wissenskombinat Guild
  • Marketing: Outlook for the 1st half year of 2015
  • Trainee @ comSysto + small workshop
  • Managing directors table
  • Introducing the new office team and their roles
  • GULP 2.0

The topic tables at comSysto were similar to a booth at an exhibition, and each topic table covered a different subject matter. Each colleague had the chance to drop by and inform themselves about the certain focus area. Many of the topics were work related. As most of the talks have lots of internal information about clients and projects, we can only show you two of the sessions (GULP 2.0 and JVM Deep Dive).

The guild tables were groups of employees that share knowledge, tools, codes about certain topics. In the BIG PICTURE Guild, employees explore data in small projects like Kaggle competitions, sensor data analysis, IOT, location tracking and anomaly detection. They basically try to get knowledge out of the data mainly by using machine learning methods. Wissenkombinat is the guild that has its focus on knowledge transfer and employee development. The aim of the Wissenkombinat is to improve the good “knowledge” (e.g. increasing efficiency, increasing communication between each other and transferring knowledge, etc.) and to find ways how to better learn from each other. If you want to read more about our guilds (there are several more) then please follow this link to our website.

Featured image        Featured imageFeatured image         Featured image

Each attendee had the chance to rate the talks, workshops and topic tables with our mascot sticker (see below).


Work Hard, Play Hard

Featured image      Featured image

The chillout corner was very popular. comSysto employees had the chance to play playstation, which by the way belongs to teambuilding :).

What do you think about comSysto’s One Conference?

Does it encourage you to start your own internal conference?

You are so convinced by it and want to join our team? 🙂

Share your thoughts with me.

Build Light – Continuous Delivery meets Reengineering an USB driver

Continuous Delivery is a pattern language used in software development to automate and improve the process of software delivery. Techniques such as automated testing, continuous integration and continuous deployment allow software to be developed to a high standard, easily packaged and deployed to test environments. Continue reading

DevOpsDays London 2013 Roundup

I had the pleasure to visit this year’s DevOpsDays Conference in London. DevOpsDays is still a very small conference very much focused on communication between participants. This means that about half of the time is assigned to so called “Open Space” sessions. These sessions allow all attendees to suggest topics that are import to them which can then be discussed with whoever is interested in the topic. Continue reading

Velocity Europe 2012 Roundup

This year’s Velocity Europe took place in London from October 2nd to October 4th. It had two main topics: “Web Performance” as well as “Operations and Culture”. The first day was a warm up day sporting longer more interactive sessions.

In summary the Web Performance talks were quite interesting. From learning some new tricks about Chrome Dev Tools from Google’s brilliant Ilya Grigorik to getting an outlook on HTTP 2.0 by Akamai’s Mark Nottingham. It seems that some current best practises on operating web applications might change in the future. Continue reading

Continuous Delivery Of Waste or How We Started Thinking About What To Build

It’s been a while since I have first started looking for relevant resources about how we can find “a right thing to build” on our own. Almost every geek friend I personally have and almost any colleague of mine at comSysto dreams about building their own product at some point, not necessarily in order to earn a few million (though no one would actually complain about that minor side effect) but rather in order to be able to decide on his or her own what approach and frameworks are to be used and what features are to be built. Continue reading