Speaker: Lynn Owens

For more, see the table of contents.

---

Intro/Problem

• Every gitlab project has own .gitlab-ci.yml file. Great for getting started
• Quickly have hundreds of projects
• Goal is to eliminate copy/paste by centralizing in a few projects

What NAIC has

• 200+ projects maintained by 11 teams in 2 dev orgs
• Pipeline is inner source
• Version 6 of pipeline; working on version 7
• Reduced maintenance burden by making change once and not in each project
• Hosted directly on gitlab.com

Milestone 1 – Hidden jobs for pipeline project

• GitLab has “hidden” jobs
• Start with a period
• Don’t appear in any pipeline; just for the common code
• The “pipeline” project has a .gitlab-ci-base.yml which contains common code
• Common code makes no assumptions about teams and is configurable for all known use cases
• v1 was about two dozen lines of common code
• The client projects include the pipeline code (can include in any part of gitlab so doesn’t need to be yours)

include:
   -project: 'NAIC/pipeline' 
   -file './gitlab-ci-base.yml'

• Then added jobs that extended the hidden jobs to call functions in the base code. Where deploy_foo is in the base code

deploy_foo:
  stage: deploy
  extend: .deploy_s3
  variables:
   ...

Suggested practices

• Advises against pinning the pipeline to a tag because don’t get bug fixes and everyone has to upgrade manually
• Don’t include stages in the pipeline as it forces one opinion on everyone. Many groups had written a pipeline for their use case and not all same.

Milestone 2 – Profiles

• Found a half dozen use cases. ex: Maven for Java, NPM building Angular etc.
• The .gitlab-ci.yaml was a copy/paste of the others in the use case.
• Made profiles/maven-java.yml and the like in the common profile
• Profiles are not one size fits all because there are a bunch of different ones and can still use the milestone v1 approach.

Milestone 3 – Pipeline scripts

• Common code like logging, calling rest apis, etc
• Switched from bash scripting to python so had common code in modules and could unit test the modules

Options to get scripts

• Could have the pipeline create a tar.zip and upload to a repo. This is a little slow
• Could have a global before_script that does a git clone of peipleine-scripts. Uses a network connection
• Could bake the scripts into an image. Requires a pipeline

If was doing again, wouldn’t create separate pipeline-scripts because tightly coupled to pipeline. Doesn’t change problem of using the scripts though.

Testing

• If client projects are all using the default branch, small changes will affect them all.
• Use a testing framework for script code (ex: python/go)
• Follow development practices
• Write a sample app for each profile. Have the common pipeline trigger a downstream pipeline on this project. For any merge to master, the downstream jobs must pass.
• Before major refactors, inventory profile jobs and audit afterwards,

Milestone 4 – Profile Fragments

• Had about 24 profiles (ex: maven-java-jar, maven-java-pom, maven-java-k8s, etc)
• Typically three components – build tool, language, deployment method
• These profiles had a lot of copy/paste
• Decomposed into fragments – ex: maven, npm, java, angular, k8s, s3)

Selling the idea

• Needed to convince people to use this pipeline instead of writing own or another team.
• Offer flexibility
• Show value
• Follow semantic versioning to the T (he tags every merge to master of the pipeline even though encourages use of the default branch. the tags are good rollback points or if the project needs something older)
• Changelog everything
• Document well
• Train and evangelize
• Record training so have library

My take

This was a good case study and useful to see concrete examples and techniques. I wish we could see the code, but I understand that belongs to their org.