Lessons learned from my CICD journey in work

World is quite simple when you doing things yourself and always getting easier day by day. basically you code you push , build then deploy. that’s it.

The core logic can be wrapped up in a simple yaml file more like this

jobs:
  my_ci_cd:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@master

    - name: build
      uses: peaceiris/actions-hugo
      with:
        args: --gc --minify --cleanDestinationDir

    - name: deploy
      uses: actions/aws/cli@master
      with:
        args: s3 sync ./public s3://yous3bucket --delete
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

That’s right , above example shows a content code built with hugo site generator then deployed to S3 . It is actually an example of github action. Before it you might used TravisCI ,CircleCI or Drone.io.

Well all those things are common in :

• A serverless CICD server and agents runs all sorts of build and deploy
• Container based build system with dependency managed by container image.
• A thin and simple pipeline description , mainly a single yaml file.

Why all those CICD solutions are not being used by corporate project?

Serverless is not welcomed by enterprise CICD

All my previous workplace adopted AWS quite a while. Some of them even approved to use AWS Lambda to develop their microservices. However CICD pipeline never goes down to this path , in my opinion security compliance of most corporate is the main reason

• Public CICD service providers are lacking of region/country based service model
• Data flow doesn’t support self encryption mechanism, unlike most cloud provider

I guess people might know buildkite , which brings a new idea by shifting agent to corp site and leave server as serverless. I personal endorse this creative thinking , however server agent communication and build log output flow is still a concern for some enterprise customer.

Jenkins

is still favoured by most of enterprise developer especially Java developer.

If you look at enterprise pipeline in particular jenkins pipeline. It is quite extraordinary within a big organisation:

• Different version of Jenkins server with different plugin version associated with it and everything is plugin.
• Different form of pipeline from scripted to declarative.
• Pipeline share library (groovy step or stage)

With all above things be applied together , we running into a few quite popular problems

• I want to upgrade Jenkins .No ,the plugin can’t be lifted and it will not compatible with new version Jenkins.
• I want to change my pipeline. No, you have to PR to share library which maintain by other team.
• I don’t like groovy as pipeline language , can I use yaml . No , unless you use jenkinsX , which basically is not the jenkins you think of as before.

A Small experiment in my project

A project I was involved want to try something different but one thing has to be same: use Jenkins!

Long story short, we made some changes on classic way jenkins pipeline

• Not reply on any Jenkins plugin in any step.
• Use dockerfile or docker image as agent

agent {
   filename 'Dockerfile'
}
steps {
   sh -c "do something"
}

With those little help I managed to get same code build in same time with another CICD tool GoCD . Actually it will be same for those Dockerfile convert to github action used even in serverless CICD without affected your build and deploy script.

And Jenkins ‘s creator seems have same feeling with me

https://www.previous.cloudbees.com/blog/move-toward-next-generation-pipelines

Next Generation Pipeline is defining a simple yaml syntax to define abstract Pipelines, without exposing implementation details

pipelineConfig:
  pipelines:
    release:
      pipeline:
        agent:
          image: nodejs
        stages:
          - name: Build
            steps:
              - command: echo
                args:
                  - hello world

Wow, a serverless Jenkins pipeline as next gen pipeline. To me it looks familiar, can’t remember where I see this pipeline before 😅

Things went too far

Cloud is booming and so does your pipeline. With help of all kinds of cloud provisioning , enterprise wrapping up jenkins pipeline libs and plugins as template quickly pushed to organisations. Many Jenkins instances being created, even more pipeline are created under this pattern, while jenkins itself moves to docker based serverless model.

A few things I would suggest even continue to use Jenkins

• Reply docker rather plugin , I proved it works while moving to different CICD solution without cost too much work. And docker image can be share just like the pipeline libs and plugins.
• Make pipeline simple. Pipeline is not code,down play it ,move complex logic into deploy script.

More thoughts:

• Serveless CICD will come to enterprise, and it just a runner , you don’t care the version.
• Migration can start from now as all we need to containerize your pipeline runner dependency, it still works even in your current model.
• Invest on Gitlab CI or Drone or other similar tool as self hosted solution should be right direction. If Github can bring githubaction into its self hosted enterprise version it will make this adoption more promising.
• I feels a decent docker registry somewhere missing. Doesn’t like AWS ECR, Artifactory etc.