Menu
Andreas Joelsson

Introduction

TL/DR: Introduction and why we ended up with GitLab.

In our project to replace our Jenkins monolith build system running on old retired hardware, we evaluated a lot of options. One of the requirements was to run each job/task as a container so we evaluated tools like Drone, Jenkins X, Gitkube, Concourse, GoCD, Argo and GitLab.

With requirements to have it file-drive (setup as code), trigger on branches, tags and PR and it should be running inside kubernetes. And the nice to have was generic webhook support, persistent storage for caching tools and something like or similar to an org-scanner.

We decided finally on GitLab as we liked the close integration between our repositories with merge request and CI/CD pipelines all in one place. Sure there are a lot of more features there that we are not using, where options to enable/disable features in the menus would be nice to have (cough GitLabcough).

Migrating to GitLab from GitHub.

TL/DR:Migrating from GitLab to GitHub, pitfalls and tips.

There are a lot of good features where you can keep repositories in sync to make the transfer as seamless as possible. But our experience is that you should aim for one way transfer so when you move, you should not update the old repositories anymore. Even if there are sync options they are not as flexible as you might think and you will hand over all the merge issues found to a robot. Not a good idea in our experience.

Building the pipelines

TL/DR:Introduction to GitLab pipelines yaml, changing the behaviour of merge request pipelines.

Building pipelines in GitLab is quite straightforward using command line tools and yaml. There are a lot of experimental features and odd things that you need to wrap your head around. And a black belt in Bash scripting isn't too bad either.

One of the key things we wanted was to change the behaviour of the merge request depending on what labels you applied to the merge request.

This required us to try and split and share the ci-templates in specific areas, where some were shared between code bases and some were specific to the language of that code base. We have deployments, IDL and code quality as shared ci-templates for all languages while there are a number of specific templates that is language specific. We try to keep the same stage names for language specific so any two pipelines has the same stages while different jobs is run under them.

How to adapt the merge request pipeline.

TL/DR:Example and YAML to change the merge request pipelines.

One of the crucial things to make a YAML job work to trigger changed behaviour from triggering labels is that it needs to have only: label for merge_requests like:

verify:jdk11:
  <<: *verify
  artifacts:
    expire_in: 12 hour
    paths:
      - target/*.jar
  only:
    - merge_requests
    - master

The reason for this is that in the job to be able to trigger on special labels you need to build the pipeline on the $CI_MERGE_REQUEST_LABELS variable. This is only available in the special branch merge_requests. Which makes sense as there are no labels on a branch, but only on merge requests. Here is an example job that only is triggered if we apply the test:deploy label.

test:pre-release:
  <<: *test
  only:
    refs:
      - merge_requests
    variables:
      - $CI_MERGE_REQUEST_LABELS =~ /test:deploy/

So this part will never be executed unless we apply the test:deploy label. And also the label needs to be set before the pipeline is built, otherwise, it will not trigger on it. There is no way to trigger a pipeline today in GitLab when we add or remove labels. This could be automatic after they have added the suggested issue or could be implemented by the bot to trigger after it has applied the labels.

NML96eA.png#asset:438

Applying the merge request label test:deploy.

WPHB8gb.png#asset:439

Will alter the pipeline so that the merge request will deploy it directly to our test environment.

qygB2bm.png#asset:440

This enables developer besides me to roll out features and test them out as you develop. This is useful for bigger features where you want to integrate with other clients before release.

I myself usually do this command line but there is little to no overhead compared to using the pipeline so we have taken it so far that it enables me and my colleagues that built the CI/CD to use the pipeline over command line deployment to kubernetes.

How to change the release branch

TL/DR: Examples and how to change the release branch from a merge request.

So, the million dollar question. We have managed to customize our merge request pipeline using the variables provided in the GitLab environment where we can catch the labels if we are in a merge request.

The problem with the release branch or master branch is that once the merge request is done, no information from the merge request is available in the master pipeline except the commit message. This is what we should be able to trigger on information like the upstream labels or something similar. Sadly this is not the case as of today.

So what is the solution then? The one we have come up with and is rather crude is to make use of the commit message to trigger behaviour of the master pipeline using a GitLab bot triggering on updates to the merge requests.

Todays Crude Solution

So the crude solutions employ a GitLab bot that updates the title of the merge request depending on what label is being applied to the pipeline.

8SHLSK4.png#asset:441

So the webhook triggers when a merge request gets or removes a label and adds them to the title of the merge request.

This is because when the merge request is being merged in GitLab we can trigger pipeline behaviour using the $CI_COMMIT_MESSAGE to build up the master pipeline to what we want using the same methods as on a merge request with only and except. The example YAML file for the semantic versioning is below, feel free to skip this as it is really technical.

#
# Template for tagging, Takes a variable for BUMP_MODE that can be patch, minor, major or hotfix.
#
.git-tag: &git-tag
  image: ${Z_GIT_IMAGE}
  stage: release
  tags:
    - zedge
  except:
    refs:
      - tags
    variables:
      - $CI_COMMIT_MESSAGE =~ /\[deploy:skip\]/
  before_script:
    - ... hackish way to set up ssh credentials.
    - export OLD_TAG=$(git-latest-tag)
    - export NEW_TAG=$(git-semver -i ${BUMP_MODE} ${OLD_TAG})
  script:
    - |
      echo "ChangeLog: ${NEW_TAG} -- $(date '+%Y-%m-%d %H:%M:%S')" > .__changelog.txt
      echo >> .__changelog.txt
      if [[ "${OLD_TAG}" != "v0.0.0" ]]
      then
        git log --no-merges --decorate= --oneline --shortstat ${OLD_TAG}.. >> .__changelog.txt
      else
        git log --no-merges --decorate= --oneline --shortstat >> .__changelog.txt
      fi
    - |
    - git tag ${NEW_TAG} -F .__changelog.txt
    - git push origin ${NEW_TAG}

#
# Will create a major release tag.
#
git-tag:major:
  <<: *git-tag
  variables:
    BUMP_MODE: "major"
  only:
    refs:
      - master
    variables:
      - $CI_COMMIT_MESSAGE =~ /\[bump:major\]/
  except:
    variables:
      - $CI_COMMIT_MESSAGE =~ /\[deploy:skip\]/

#
# Will create a minor release tag.
#
git-tag:minor:
  <<: *git-tag
  variables:
    BUMP_MODE: "minor"
  only:
    refs:
      - master
    variables:
      - $CI_COMMIT_MESSAGE =~ /\[bump:minor\]/
  except:
    variables:
      - $CI_COMMIT_MESSAGE =~ /\[deploy:skip\]/

#
# Will create a patch release tag. This is the default tagging.
#
git-tag:patch:
  <<: *git-tag
  variables:
    BUMP_MODE: "patch"
  only:
    - master
  except:
    variables:
      - $CI_COMMIT_MESSAGE =~ /\[bump:major\]/
      - $CI_COMMIT_MESSAGE =~ /\[bump:minor\]/
      - $CI_COMMIT_MESSAGE =~ /\[deploy:skip\]/

Here we see how you can modify the semantic versioning of the release branch from the merge request (except some ssh setup and credentials to git). It always defaults to a patch level release so every release to master has a tag and can be rolled back to. And in the merge request you can as part of the review process decide in our case:

  • We add features or changes to the interfaces, we apply minor release.
  • We break the current API / remove features from it, we apply a major release.
Now there are other reasons we bump minor or major version as well but this is the most common that we try to enforce.

The flakiness of this handling is that if you apply a label and then click Merge. Nothing will happen. Because you need to actually refresh the page to get the updated title before you merge or nothing will happen.

Example: Semantic Versioning

The first thing we implemented was the semantic versioning where all the releases create a patch level. Unless you in the review process apply a minor or major release to that merge request. So in the label part, you have 4 options to select between.

spDXfN5.png#asset:451

If nothing else is selected the patch is applied by default. Also, the gitlab bot makes sure that the latest always is selected so if you change from minor to major it will remove the [bump:minor] from the title and apply the [bump:major].

A Hotfix is a special case we have added where it creates a pre-release tag and takes the MR into production with fewest steps possible. That means building without test, create docker image and tag as pre-release, push to production and deploy it. Usually what I do command line, but this gives all the developers this power, even though you might not be comfortable with kubectl and command line.

So depending on what label is applied (or not applied) the release step will take the following three forms.

kLhxGWq.jpg#asset:453

Thus you get the release functionality integrated in the merge request which is IMO the most flexible way where you can do it. As your colleagues can give feedback to the mr if you add or remove functionality as well.

Example: Optional<Continous Deployment>

The other example that much of the post is about is to enable the developers to do continuous deployment. The options we provide are manual, auto and skip. The default setting is manual where you press a play button. But this can be changed on a repository level to have the default behaviour of automatic deployment. But you still get the option to go back to a manual deployment. The skipped part is things like documentation updates, or increase test coverage. You just don't want to do a new docker image that is identical to the last one, nor roll it out in test or production.

The label options are the following:

aWHnbTY.png#asset:454

Not showing the manual as we for some reason named it manual instead of prod:manual so they were not next to each other. This will alter the release part in two ways (skip won't do anything), the default behaviour:

CRfLgLd.png#asset:455

And the automatic rollout:

AGHCDjV.png#asset:456

The gitops downstream is exactly the same, the only difference is that one is automatic and the other is a manual interaction as an extra step.

And then?

What you might come up with is up to you, but this gives endless possibilities but you need to keep in mind that it can get quite complex, so templating is a strong suggestion when going into this. We are also using multiple repositories for our services so I cannot tell how this will work with a single (Google way) repository.

Some of the things we have planned for the future is:

  • Move the deployment part to the tag, including the service-specific config, so rollbacks can be done from the tags in gitlab.
  • Add/remove more or less testing/integration test using labels deciding what to test and what not to test as well to increase the speed of the pipelines.

But Why

TL/DR: Why should you do this, what's to gain.

One of the main reason for adding this functionality is to enable our developers to take the leap into continuous deployment. Meaning that we provide a default option where you roll out to production using a play button. But can in the merge request change this to automatic rollout, or skip rollout all together (read document update f.e.). This puts the CD decision in the hands of the developer and will i.m.o help people to adopt the new CD concepts. My team has started to add CD as a default option on some projects with the option to turn it back to a manual step if you feel like it.

This was also something I discussed a lot with CI/CD people during the latest kubecon in Barcelona, where most did not have any solution for this in place. Closest was prow but they were more on the merge request level as I understood it compared to altering the master branch as we want to achieve.

I was on the CDF workshop during KubeCon and a lot of the talks were about culture (Sadly the pre-workshop is not available on YouTube) to adopt CI/CD to always roll out in production. I feel this approach will leave around 90% of companies not taking the plunge into the CD world, but an optional approach that this is giving will get more people to try it out. Because you can force culture onto someone or you can do it in a way you feel comfortable with or let a senior engineer take the decision applying the prod:auto to a merge request.

This will build the confidence in the single engineer to push more to production as well as given the option to halt for a while and build confidence testing. Without having a test suite google would be proud of.

What can you do for the CI/CD community

TL/DR: How to make this more robust / standardise the way of doing this to all git providers.

  • We have a feature request @GitLab for a more stable solution where the default branch can get $CI_XXX with the labels that were provided in the merge request. Help bump this up so they consider it as a feature.
  • Try to influence/poke CDF to create an API standard that can build custom pipelines downstream that can be implemented by any git provider.
  • Get Tekton to consider this as part of their build for CRD for CI/CD as a way to alter the pipeline pre-run compared to alter it depending on what happens during the run.

Next steps

  • We are gonna open source our GitLab bot that updates the title from merge request label events.
  • We are gonna open source one of our repositories that provides these features that you can clone/try out asap.