Jenkins vs Azure DevOps (formerly known as VSTS)

Jenkins is probably the number #1 Continuous Integration (and Continuous Delivery) tool for Java developers. It is very flexible and has a lot of supported plugins. There are more than 3000 plugins, so plenty to choose. This flexibility however also comes with a price. If you want to use Jenkins in a serious pipeline, you need a scripted workflow, fully integrated with other tools, secure from begin to end, etc. This is doable, but it is frustrating to implement and it certainly costs a lot of time.

For example, a lot of plugins depend on other plugins and often these need to be very specific versions. This results in a "plugin dependency hell". Also the use of some plugins is not always very straightforward. This certainly applies if you want to use them in a scripted pipeline. The examples on the Internet are not always clear and some descriptions assume that you know the ins- and outs of Jenkins. In addition, not all plugins support a multibranch pipeline, which basically makes them useless in modern CI/CD.
Statistics and metrics are non-existing in Jenkins, so if you are into these things you need a couple of plugins and Grafana to visualize, but you have to set this up from scratch.

Lately I have been working with Azure DevOps, formerly know as VSTS (Visual Studio Team Services) and although I am a bit biased when it concerns Jenkins (I do love it, really) and the combination of Microsoft and Java has not always been a good marriage, Azure DevOps grows on you. And Azure DevOps can be used for Java development. It supports Maven and Gradle tasks and the agents are able to run on Windows (of course), Mac and Linux. It has a lot of build-in tasks, both from Microsoft itself, but also downloadable tasks from the Azure DevOps marketplace. And if a task is not available you are able to create it yourself (using typescript).

I managed to set up a small pipeline in a day. This included a Maven build, SonarQube and Fortify scans and a deployment to PCF (Pivotal Cloud Foundry). Some Azure DevOps task are very powerful. I needed something to replace secure credentials in a manifest.yml file. No problem. I added a Tokenizer task from the Marketplace, entered the secure credentials as variables and that was it. I've done this in Jenkins before, but that took me way more time.

Does this mean that I abandon Jenkins? No, certainly not. Azure DevOps is not a silver bullet and some things are not very well supported or need to be improved. A small comparison:

Jenkins	Azure DevOps
Very flexible, scriptable workflow	Workflow is limited and straightforward (no real if-then-else or switch-case constructions; only some simple filtering and conditions can be set). This makes it more difficult to develop complex workflows
Pipelines must be programmed (in Groovy)	Construction by means of a GUI is much easier and faster
User interface is OK, but limited	User interface is confusing
A lot of plugins, although some are not very useful, not straightforward or unstable	A lot of powerful tasks out-of-the-box; easy to use
Plugin dependency hell	NA
Redundancy of code in a pipeline script is not needed	Duplication of tasks (because of the first point), although this can be solved by means of task groups
Secret credentials and files support	Secret credentials and files support
Open source and support by Cloudbees	Closed source and support by Microsoft; a team of about 400 developers are working on this product
Not provided as SaaS solution	Provided as SaaS solution
Does not include workitem management, Git, or functionality to deploy to Cloud	Full solution, including workitem management, Git, easy deployment to Cloud (not only Azure, but also AWS)
Metrics/statistics are non-existing. No configurable dashboard	All steps in the process are registered. Azure DevOps has support to create custom Dashboards, although the number of widgets is still limited
Traceability of changes in the pipeline are only tracked if the pipeline is in Git. Traceablility of a build and release has to be programmed in the pipeline	Changes in the pipeline are registered (every change is a commit). All steps in a build and release are traceable out-of-the-box
Secure, but some things can be by-passed (eg. the "Replay"  function makes it possible to make changes in the pipeline or even extract credentials)	Secure
NA	No support for Webhooks if you use an external Git repository
Notification possible	Notification out-of-the-box

So, the question is "who is the winner ?". I think it is a tie. Jenkins is very flexible when it comes to creating complex workflows, but it is faster to get something working in Azure DevOps. In fact, I have a CI/CD pipeline that contains a combination of both. I use DevOps Azure for the main build and deployment (release) tasks and Jenkins as an API to disclose automated tests (Jenkins is called from a Azure DevOps release flow). In addition we still have Jenkins jobs which are not (yet) ported to Azure DevOps, because of time contraints; e.g. we use the Swagger-Confluence CLI (https://cloud.slkdev.net/swagger-confluence/) + some Groovy script and Confluence REST calls to generate documentation. There are no Azure DevOps marketplace tasks yet, so porting it is not straightforward.

Workflow

Workflow is an important feature of a CI/CD pipeline. Some people advocate that only one workflow option is the correct one and that is "CI running against a shared mainline". Your branch strategy looks something like this:

This means that all changes are committed into one Master branch and every commit is a potential release candidate. The CI/CD actions are:

Branch	CI/CD actions
Master	Perform full build and Unit tests Quality checks / static code analysis Deployment and testing Production deployment

Promotion and deployment to Production can be a full automated process, but it may be convenient to insert a manual step.
The workflow is simple, but it may not be suitable for all teams. Sometimes it is not easy - or possible - to split up a feature into small changes. Frontend applications often involve small GUI changes, so a workflow that consist of only a Master branch may be sufficient, but in a lot of cases, a feature must be implemented as one change. A branch-per-feature workflow may be more suitable in these cases.
Some literature state that other workflows are considered 'CI/CD theatre' (see also https://www.thoughtworks.com/radar/techniques/ci-theatre). I find this a bit debatable. You must choose the workflow that the team feels comfortable with and that makes sense for the type of changes. If your workflow has some sort of mainline you are still be able to "run CI against a shared mainline". In our case for example, we have a Java/Maven project using the Atlassian workflow; this means the use of a branch-per-feature. The workflow is as follows:

The Master branch is the mainline. Each user story or epic results in a Feature branch. Each successfully build Feature branch is merged with the Master. A failed Feature - Epic 1 in the figure - is not merged with the Master until the build and unit tests are successful. Each Release is a branch with a commit from the Master. In CI/CD pipeline terms this means the following actions for each branch type:

Branch	CI/CD actions
Master (=snapshot)	Perform full build and Unit tests Quality checks / static code analysis Deployment and testing
Feature	Perform full build and Unit tests
Release	Perform full build and Unit tests Quality checks / static code analysis Deployment and testing Production deployment

Compared with the first workflow, the branch-per-feature workflow has one distinct difference; the commit frequency on the Main branch is less. That does not have to be an issue. The commit frequency on the Feature branch may be high instead. A merge with Master also does not mean that you easily screw up the Master branch. It is the responsibility of the developer to keep the Feature branch in sync with the Master.
The branch-per-feature workflow also does not put a heavy burden on the build- and test environments. Even the smallest change may result in a long build- and test cycle (a Fortify scan can easiliy take up 1 - 2 hours) in a one-branch workflow . Making a lot of small changes per day in a one-branch workflow results in queued jobs and queued testruns, unless your build- and test environments are scaled accordingly, which becomes costly.