For example, a lot of plugins depend on other plugins and often these need to be very specific versions. This results in a "plugin dependency hell". Also the use of some plugins is not always very straightforward. This certainly applies if you want to use them in a scripted pipeline. The examples on the Internet are not always clear and some descriptions assume that you know the ins- and outs of Jenkins. In addition, not all plugins support a multibranch pipeline, which basically makes them useless in modern CI/CD.
Statistics and metrics are non-existing in Jenkins, so if you are into these things you need a couple of plugins and Grafana to visualize, but you have to set this up from scratch.
Lately I have been working with Azure DevOps, formerly know as VSTS (Visual Studio Team Services) and although I am a bit biased when it concerns Jenkins (I do love it, really) and the combination of Microsoft and Java has not always been a good marriage, Azure DevOps grows on you. And Azure DevOps can be used for Java development. It supports Maven and Gradle tasks and the agents are able to run on Windows (of course), Mac and Linux. It has a lot of build-in tasks, both from Microsoft itself, but also downloadable tasks from the Azure DevOps marketplace. And if a task is not available you are able to create it yourself (using typescript).
I managed to set up a small pipeline in a day. This included a Maven build, SonarQube and Fortify scans and a deployment to PCF (Pivotal Cloud Foundry). Some Azure DevOps task are very powerful. I needed something to replace secure credentials in a manifest.yml file. No problem. I added a Tokenizer task from the Marketplace, entered the secure credentials as variables and that was it. I've done this in Jenkins before, but that took me way more time.
Does this mean that I abandon Jenkins? No, certainly not. Azure DevOps is not a silver bullet and some things are not very well supported or need to be improved. A small comparison:
| Jenkins | Azure DevOps |
|---|---|
| Very flexible, scriptable workflow | Workflow is limited and straightforward (no real if-then-else or switch-case constructions; only some simple filtering and conditions can be set). This makes it more difficult to develop complex workflows |
| Pipelines must be programmed (in Groovy) | Construction by means of a GUI is much easier and faster |
| User interface is OK, but limited | User interface is confusing |
| A lot of plugins, although some are not very useful, not straightforward or unstable | A lot of powerful tasks out-of-the-box; easy to use |
| Plugin dependency hell | NA |
| Redundancy of code in a pipeline script is not needed | Duplication of tasks (because of the first point), although this can be solved by means of task groups |
| Secret credentials and files support | Secret credentials and files support |
| Open source and support by Cloudbees | Closed source and support by Microsoft; a team of about 400 developers are working on this product |
| Not provided as SaaS solution | Provided as SaaS solution |
| Does not include workitem management, Git, or functionality to deploy to Cloud | Full solution, including workitem management, Git, easy deployment to Cloud (not only Azure, but also AWS) |
| Metrics/statistics are non-existing. No configurable dashboard | All steps in the process are registered. Azure DevOps has support to create custom Dashboards, although the number of widgets is still limited |
| Traceability of changes in the pipeline are only tracked if the pipeline is in Git. Traceablility of a build and release has to be programmed in the pipeline | Changes in the pipeline are registered (every change is a commit). All steps in a build and release are traceable out-of-the-box |
| Secure, but some things can be by-passed (eg. the "Replay" function makes it possible to make changes in the pipeline or even extract credentials) | Secure |
| NA | No support for Webhooks if you use an external Git repository |
| Notification possible | Notification out-of-the-box |
So, the question is "who is the winner ?". I think it is a tie. Jenkins is very flexible when it comes to creating complex workflows, but it is faster to get something working in Azure DevOps. In fact, I have a CI/CD pipeline that contains a combination of both. I use DevOps Azure for the main build and deployment (release) tasks and Jenkins as an API to disclose automated tests (Jenkins is called from a Azure DevOps release flow). In addition we still have Jenkins jobs which are not (yet) ported to Azure DevOps, because of time contraints; e.g. we use the Swagger-Confluence CLI (https://cloud.slkdev.net/swagger-confluence/) + some Groovy script and Confluence REST calls to generate documentation. There are no Azure DevOps marketplace tasks yet, so porting it is not straightforward.
