Microservices: CI with LambdaCD - Current architecture and vision for the future (3/3)

Over the time we added pipelines for new services to existing LambdaCD instances because the services are very small and we make changes to them very rarely. If we had not done that, we would have had many instances which would have run the whole day and consume resources unnecessarily.

All LambdaCD instances are based on the tesla-microservice which uses the component framework and provides some nice features like health check, status page, metrics, loading configuration files and a graceful shutdown. It is a good starting point if you don't want to reinvent the wheel every time. Check it out! Maybe you can use it to create your own microservices or you can make a fork to adapt it to your needs.

Current Architecture - Build-Monitor

Our pipelines use the LambdaCD cctray extension The cctray.xml format is widely used by CI tools to report the state of pipelines and their steps. At first, we tried out nevergreen which only displays running and sick steps. But this open-source tool has two disadvantages: You can only display one cctray and the configuration is stored in the browser cache. The second point is impractical because every developer has to configure the monitor on his local machine.

After a little research we decided to implement our own solution. It is a simple Javascript application with a server-side configuration file which lets you define as many pipelines as you want. We use Flexbox to have a dynamic alignment which you can see in the left part of the picture below.

At the moment, the monitor isn't perfect because the font isn't scaled dynamically and if you have too many running or sick pipelines your screen could be too small. But I will publish it within the next weeks and maybe somebody has a good idea how to fix these problems.

Current Architecture - Graphite

To know which LambdaCD instance needs more or less resources we use Graphite With a quick view you can detect instances running out of memory, disk space or CPU power.

These graphs are very good to scale a pipeline corresponding to its consumption. Moreover, if you look at the spikes of these graphs you can see how often a pipeline is triggered or restarted.

Current Architecture - Disadvantages
The current architecture is grown over the last weeks and we collect a few things for improvement:

• Stateless applications - All of our microservices are stateless. As a result, a LambdaCD instance looses all information about past pipeline iterations.
• Many web UIs - Each pipeline has its own web UI and if you want to see why a step failed you have to know the right LambdaCD instance URL. Better would be a central service which collects all this information and displays a clear overview.
• Idle pipelines - Most pipelines are only waiting the whole day for the next run but they are still blocking resources. This isn't a problem for a few pipelines but in a year, we might have 50 or 100 microservices.

The Second Approach
Over the next few weeks, I'm going to implement some new LambdaCD components to get rid of these disadvantages. I love the flexibility of LambaCD which lets me extend the tool in any direction. I can't show you how you can build the best LambdaCD infrastructure because at the moment I don't know it. But I want to give a sense how easy it is to build a system which adapts your needs and is ready for further improvements.

Let's have a look at the overview:

The Pipeline-Builder, the Pipeline-Spawner and the Pipeline-State-Controller were added and I will introduce them in the next paragraphs.

The Second Approach - Pipeline-Builder
The Pipeline-Builder is a place for all of our meta-pipelines. You don't have to create such a pipeline in every project anymore. Just add the Git URL to the trigger of the Pipeline-Builder and you are done.

If you don't use the Pipeline-Spawner you have to add a step to deploy your pipeline.

The Second Approach - Pipeline-Spawner
If the code of the project is changed the Pipeline-Spawner is triggered by this commit and it deploys the corresponding pipeline which was built by the Pipeline-Builder. In addition, the service injects the commit hash into the pipeline to guarantee it uses the right version of the code.

You don't have to use this spawner but it is good way to save resources because you run a pipeline only when you need it.

The Second Approach - Pipelines
Pipelines don't have a Git trigger anymore in front of the other steps because they already know the commit hash which was injected by the Pipeline-Spawner. The other steps are the same as in the current architecture. But a pipeline may only be executed once why it has to shutdown itself after the last step.

The Second Approach - Pipeline-State-Controller

The Pipeline-State-Controller acts as a proxy between all other components and a database, e.g. mongoDB. It stores the state of these components and it provides two different views

• Control-Center - Just a simple view to get all information about your LambdaCD cluster and to trigger manual steps: