Ruben Hakopian
Ruben Hakopian
Sep 6, 2019 6 min read

CI/CD Pipeline Automation For SaaS Applications

thumbnail for this post

Are you running a SaaS company and every new version takes longer to release? Techniques described below can be useful for any application to significantly shorten release cycle, improve product quality, and boost overall team’s productivity.

Berlioz is a Continuous Delivery SaaS, which deploys containerized and function-based applications to the cloud and sets up cloud infrastructure as a complete solution. Different customers are using various combinations of features, and it became extremely inefficient to validate and release any new version. We came up with branching and CI/CD pipeline strategies that worked well for SaaS applications.

Another problem we solved was active development efficiency. Most applications are either too complicated, have unresolvable dependencies, take too long to build or have some other reasons to run locally on the workstation. We decided to take the CI/CD pipeline one step forward, so it takes care of active development as well.

Objective

Using the methods described below in this article, we achieved:

  1. Shortened production release cycle
  2. Significant reliability improvement
  3. Accelerated development

Branching Strategy

Before getting to pipelines, git branches have to be set up right from the start. Git Branching Strategy

Master Branch

The master branch holds a validated and production-ready code. The code in this branch can be released to live production at any moment.

Staging Branch

The staging branch accumulates changes from development branches. It contains code we are willing to release to production, if and only if all tests pass. It is essential to have robust test coverage. The purpose here is to eliminate manual verification of changes and automatically promote changes to production. Although it is not as critical as in the master branch, it is best not to make direct changes to the staging branch as well.

Development Branch

You can start with one dev or multiple development branches. They are branched out from staging and merged back there whenever the feature is ready. For every new feature, new cases should be added. If it makes sense for your business, you could have another level of branches similar to GitFlow. But SaaS applications rarely need such kind of version control.

CI/CD Pipelines

We’ll go over pipelines in the reverse order: dev, staging and then production.

Development Pipeline

Any change to dev-* branches triggers this pipeline. We want to run it as quickly as possible. Your developers would depend on this pipeline to try out and experiment with current changes during active development. If your unit-tests are short-running, you can incorporate them into this pipeline as well.

It is vital to have the ability to bring up ephemeral environments as a part of the pipeline. We are eating our own dog food here. We use our current software version to deploy the next one. So our dev pipeline consists of the following steps:

$ git clone
$ run unit tests
$ berlioz deploy --region $CLOUD_REGION --deployment $BRANCH_NAME --wait

First two steps need no explanation. In the last step, we are making use of berlioz CLI tool. This one line takes care of all deployment needs:

  1. Called from the root of the code base,
  2. Identifies and builds Docker containers and lambda functions,
  3. Uploads artifacts to image repositories,
  4. Triggers deployment to environment matching our current git branch,
  5. Waits until deployment is completed.

You can learn more about the deployment command here.

In our case, it takes about 90 seconds, which is enough to stand up and have a quick leg and neck muscles stretch before moving on with the rest of the tasks.

For all this magic to work, we use git hooks that get triggered when new dev-* branches are created or deleted. From branch create hook we register new environment and associate it with the cloud account. This command does not use any resources, until deploy command is called, and merely associated the deployment with the cloud account.

$ berlioz deployment create --name $BRANCH_NAME --provider <cloud-provider-name>

Or clean it up using:

$ berlioz deployment delete --name $BRANCH_NAME

Staging Pipeline

Staging pipeline gets triggered when dev-xxx branches are merged into staging. Here we want to validate all tests. It is vital to identify customer personas and have individual test suites to cover every particular use-case. The overall pipeline should look somewhat like this:

Step 1. $ git clone
Step 2. $ run unit tests
Step 3. $ berlioz deploy --region $CLOUD_REGION --deployment "staging" --wait
Step 4. $ run test suite towards staging $CLOUD_REGION
Step 5. $ git merge staging to master

You can see the familiar steps: git clone, run unit tests, and berlioz deploy. Though this time, we are deploying to staging deployment environment instead.

At Step 4, execute tests towards the staging environment. It would be best to run multiple test suites in parallel, each representing one of the customer personas. The test suite should use endpoints corresponding regional staging environment.


Once all tests pass, we can safely merge staging into the master branch.

Production Pipeline

We are finally getting to production. This pipeline is very similar to staging pipeline. Except we deploy to two regions and execute verification before moving to the next region.

Step 1. $ git clone
Step 2. $ run unit tests
Step 3. $ berlioz deploy --region $CLOUD_REGION_1 --deployment "prod" --wait
Step 4. $ run test suite towards staging $CLOUD_REGION_1
Step 5. $ berlioz deploy --region $CLOUD_REGION_2 --deployment "prod" --wait
Step 6. $ run test suite towards staging $CLOUD_REGION_2

Since changes are coming in from the staging branch only and tests for staging branch were already passed, failures are unlikely to happen because of code changes. Any unexpected failures would most probably occur because of external dependencies, and rolling back to the previous version might cause even more unexpected behavior. Due to that reasoning, we are not doing an automatic rollback to the earlier version and instead rely on human intervention. This might be different for your use case.

Production Pipeline

There is no such kind of thing as a complex CI/CD pipeline. But there are many poorly developed pipelines, that provide zero to little information when things go south. We invest a lot into automation and break up the whole pipeline into manageable steps that help with issue resolution, provide meaningful and actionable details right inside the UI.

Verification

As you might have already guessed, verification is a crucial component of the entire pipeline. In some cases, running all verification tests from within CI/CD pipeline is not always possible. It can be due to many reasons, including pricing, instance sizes, and other runtime environment specifics. In the previous article, we described the method of running CI/CD pipeline steps in external AWS EC2 instances. For us, it meant execution complete freedom of instance choice and significant cost reduction.

comments powered by Disqus
© 2019 Rudi, Inc - All rights reserved