CI/CD pipeline notes: Jenkins, friend or foe 1/2

Jenkins is probably the most used CI/CD tool and it is indeed very powerful. On the other hand, setting up a CI/CD pipeline with Jenkins from scratch is a big and complex job. The road to reach to the level where we are now has been a bit bumpy. Lack or unclear documentation was a big part of the problem.

Installation

First thing to do was installing Jenkins. On itself, this is a very easy task, so no problem, but we made it more difficult than needed. The decision was to run Jenkins in a Docker container on a managed Docker platform. The main reason was that this would also run in the Cloud. This decision looked ok at first, but this also meant that:

We had to know Docker
We needed to gain knowledge of a managed platform to run the Docker containers (e.g. Kubernetes)
Everything else we used - such as Maven, Fortify SCA client, etc.- also had to run on Docker

It was a hassle to set things up initially but this changed over time and although at some point everything worked fine, it was still a complex stack to manage; I only wanted to manage my Jenkins instance.

After some maintenance issues I decided to change the setup and installed Jenkins on Linux on a VM server. Plain and simple and I was glad that I took this decision.

One of our CI/CD requirements is that everything is code. This applies to both the Jenkins installation and the configuration. Nothing is done manually. The Jenkins installation is defined in an Ansible playbook. This includes downloading and installation of the plugins, installing the initialization scripts and installing Maven and other tools that are used in the CI/CD pipeline. Everything is stored in either Git (the Ansible playbook, the scripts, ...) or Nexus (the Jenkins plugins, tools). An example of this playbook - including vars and tasks - is given below. This is a stripped version of an installation on RHEL.

Ansible file structure

The files mentioned below are organized in the following Git file structure:

main.yml (playbook)

The main playbook to be used in Ansible.

- hosts: jenkins
become: yes
become_method: sudo
become_user: root
tasks:

- include_vars: vars/variables.yml
- include_tasks: tasks/install_jenkins.yml
- include_tasks: tasks/install_jenkins_plugins.yml

variables.yml

Like it says; the variables used in the Jenkins tasks. Note, that it also refers to an rpm file which is stored in Git, together with the other Ansible files. Usually binary files are not stored in Git, but this was because of lazyness.

jenkins:
key: "jenkins.io.key"
file: "jenkins-2.127-1.1.noarch.rpm"
sysconfig: /etc/sysconfig/jenkins
plugins: /var/lib/jenkins/plugins
hook_init: /var/lib/jenkins/init.groovy.d

nexus:
url: "https://nexus3.mycompany.nl/repository/jenkins-plugins/download/plugins"

install_jenkins.yml

The main task that installs Jenkins. Note, that normally installation involves the Jenkins GUI. To circumvent this, the file /etc/sysconfig/jenkins is changed in such a way that it disables the setup wizzard.

- name: Copy Jenkins rpm package and key
copy:
src: "{{ item }}"
dest: /tmp
owner: root
group: root
with_items:
- jenkins/{{ jenkins.key }}
- jenkins/{{ jenkins.file }}

- name: Import key
rpm_key:
state: present
key: /tmp/{{ jenkins.key }}

- name: Install Jenkins
yum:
name: /tmp/{{ jenkins.file }}
state: present

- name: Set Jenkins JENKINS_JAVA_OPTIONS; disable initial setup by means of the GUI
replace:
path: "{{ jenkins.sysconfig }}"
regexp: "-Djava.awt.headless=true"
replace: "-Djava.awt.headless=true -Djenkins.install.runSetupWizard=false"

- name: Creates directory (if not existing)
file:
path: "{{ jenkins.hook_init }}"
state: directory
owner: jenkins
group: jenkins

- name: Install init .groovy scripts
template:
src: "{{ item }}"
dest: "{{ jenkins.hook_init }}"
with_fileglob:
- files/jenkins/init.groovy.d/*

install_jenkins_plugins.yml

The task that installs the plugins

- name: Copy the plugins
get_url:
url: "{{ nexus.url }}/{{ item.plugin }}/{{ item.version }}/{{ item.plugin }}.hpi"
dest: "{{ jenkins.plugins }}/{{ item.plugin }}.hpi"
owner: jenkins
group: jenkins

with_items:
- { plugin: apache-httpcomponents-client-4-api, version: '4.5.3-2.0' }
- { plugin: authentication-tokens, version: '1.3' }
- { plugin: blueocean-autofavorite, version: '1.0.0' }
- { plugin: cloudbees-bitbucket-branch-source, version: '2.2.5' }
- { plugin: blueocean-bitbucket-pipeline, version: '1.3.1' }
- { plugin: blueocean, version: '1.3.1' }
- { plugin: blueocean-pipeline-editor, version: '1.3.1' }
- { plugin: branch-api, version: '2.0.15' }
- { plugin: blueocean-commons, version: '1.3.1' }
- { plugin: blueocean-config, version: '1.3.1' }
- { plugin: credentials-binding, version: '1.13' }
- { plugin: credentials, version: '2.1.16' }
- { plugin: blueocean-dashboard, version: '1.3.1' }
- { plugin: display-url-api, version: '2.1.0' }
- { plugin: blueocean-display-url, version: '2.1.1' }
- { plugin: docker-java-api, version: '3.0.14' }
- { plugin: docker-commons, version: '1.9' }
- { plugin: docker-workflow, version: '1.14' }
- { plugin: docker-plugin, version: '1.0.4' }
- { plugin: durable-task, version: '1.15' }
- { plugin: blueocean-events, version: '1.3.1' }
- { plugin: favorite, version: '2.3.1' }
- { plugin: cloudbees-folder, version: '6.2.1' }
- { plugin: git-server, version: '1.7' }
- { plugin: blueocean-git-pipeline, version: '1.3.1' }
- { plugin: git-client, version: '2.6.0' }
- { plugin: git, version: '3.6.4' }
- { plugin: github-api, version: '1.90' }
- { plugin: github-branch-source, version: '2.2.6' }
- { plugin: blueocean-github-pipeline, version: '1.3.1' }
- { plugin: github, version: '1.28.1' }
- { plugin: htmlpublisher, version: '1.14' }
- { plugin: blueocean-jira, version: '1.3.1' }
- { plugin: jira, version: '2.4.2' }
- { plugin: jsch, version: '0.1.54.1' }
- { plugin: junit, version: '1.21' }
- { plugin: blueocean-jwt, version: '1.3.1' }
- { plugin: jackson2-api, version: '2.8.7.0' }
- { plugin: ace-editor, version: '1.1' }
- { plugin: handlebars, version: '1.1.1' }
- { plugin: momentjs, version: '1.1.1' }
- { plugin: jquery-detached, version: '1.2.1' }
- { plugin: mailer, version: '1.20' }
- { plugin: matrix-auth, version: '2.1.1' }
- { plugin: matrix-project, version: '1.12' }
- { plugin: mercurial, version: '2.2' }
- { plugin: blueocean-personalization, version: '1.3.1' }
- { plugin: workflow-aggregator, version: '2.5' }
- { plugin: pipeline-graph-analysis, version: '1.5' }
- { plugin: blueocean-pipeline-scm-api, version: '1.3.1' }
- { plugin: blueocean-pipeline-api-impl, version: '1.3.1' }
- { plugin: workflow-api, version: '2.23.1' }
- { plugin: workflow-basic-steps, version: '2.6' }
- { plugin: pipeline-build-step, version: '2.5.1' }
- { plugin: pipeline-model-definition, version: '1.2.3' }
- { plugin: pipeline-model-declarative-agent, version: '1.1.1' }
- { plugin: pipeline-model-extensions, version: '1.2.3' }
- { plugin: workflow-cps, version: '2.41' }
- { plugin: pipeline-input-step, version: '2.8' }
- { plugin: workflow-job, version: '2.15' }
- { plugin: pipeline-milestone-step, version: '1.3.1' }
- { plugin: pipeline-model-api, version: '1.2.3' }
- { plugin: workflow-multibranch, version: '2.16' }
- { plugin: workflow-durable-task-step, version: '2.17' }
- { plugin: pipeline-rest-api, version: '2.9' }
- { plugin: workflow-scm-step, version: '2.6' }
- { plugin: workflow-cps-global-lib, version: '2.9' }
- { plugin: pipeline-stage-step, version: '2.3' }
- { plugin: pipeline-stage-tags-metadata, version: '1.2.3' }
- { plugin: pipeline-stage-view, version: '2.9' }
- { plugin: workflow-step-api, version: '2.13' }
- { plugin: workflow-support, version: '2.16' }
- { plugin: plain-credentials, version: '1.4' }
- { plugin: pubsub-light, version: '1.12' }
- { plugin: blueocean-rest, version: '1.3.1' }
- { plugin: blueocean-rest-impl, version: '1.3.1' }
- { plugin: scm-api, version: '2.2.5' }
- { plugin: ssh-credentials, version: '1.13' }
- { plugin: ssh-slaves, version: '1.22' }
- { plugin: script-security, version: '1.35' }
- { plugin: swarm, version: '3.6' }
- { plugin: sse-gateway, version: '1.15' }
- { plugin: sonar, version: '2.6.1' }
- { plugin: structs, version: '1.10' }
- { plugin: token-macro, version: '2.3' }
- { plugin: variant, version: '1.1' }
- { plugin: blueocean-web, version: '1.3.1' }
- { plugin: deployit-plugin, version: '6.1.1' }
- { plugin: bouncycastle-api, version: '2.16.2' }
- { plugin: blueocean-i18n, version: '1.3.1' }
- { plugin: jquery, version: '1.12.4-0' }
- { plugin: http_request, version: '1.8.21' }
- { plugin: lockable-resources, version: '2.0' }
- { plugin: email-ext, version: '2.61' }
- { plugin: ldap, version: '1.17' }

- name: Restart Jenkins
service: name=jenkins state=restarted

- name: Wait for Jenkins to become available
wait_for: port=8080

The Ansible playbook we are using includes much more steps. It creates a logical volume, copies and decrypts credentials and uploads them into Jenkins, installs and configures specific plugins and installs tools (Maven, Fortify ...), etc..., but the structure is similar.

The next blog post explains how to configure Jenkins by means of the /init.groovy.d/ groovy scripts.

CI/CD pipeline notes

Jenkins, friend or foe 1/2