Jenkins: The Definitive Guide

Chapter 1. Introducing Jenkins

Introduction

Continuous Integration, also know as CI, is a cornerstone of modern software development. In fact it is a real game changer—when Continuous Integration is introduced into an organization, it radically alters the way teams think about the whole development process. It has the potential to enable and trigger a series of incremental process improvements, going from a simple scheduled automated build right through to continuous delivery into production. A good CI infrastructure can streamline the development process right through to deployment, help detect and fix bugs faster, provide a useful project dashboard for both developers and non-developers, and ultimately, help teams deliver more real business value to the end user. Every professional development team, no matter how small, should be practicing CI.

Continuous Integration Fundamentals

Back in the days of waterfall projects and Gantt charts, before the introduction of CI practices, development team time and energy was regularly drained in the period leading up to a release by what was known as the Integration Phase. During this phase, the code changes made by individual developers or small teams were brought together piecemeal and forged into a working product. This was hard work, sometimes involving the integration of months of conflicting changes. It was very hard to anticipate the types of issues that would crop up, and even harder to fix them, as it could involve reworking code that had been written weeks or months before. This painful process, fraught with risk and danger, often lead to significant delivery delays, unplanned costs and, as a result, unhappy clients. Continuous Integration was born to address these issues.

Continuous Integration, in its simplest form, involves a tool that monitors your version control system for changes. Whenever a change is detected, this tool automatically compiles and tests your application. If something goes wrong, the tool immediately notifies the developers so that they can fix the issue immediately.

But Continuous Integration can do much more than this. Continuous Integration can also help you keep tabs on the health of your code base, automatically monitoring code quality and code coverage metrics, and helping keep technical debt down and maintenance costs low. The publicly-visible code quality metrics can also encourage developers to take pride in the quality of their code and strive to improve it. Combined with automated end-to-end acceptance tests, CI can also act as a communication tool, publishing a clear picture of the current state of development efforts. And it can simplify and accelerate delivery by helping you automate the deployment process, letting you deploy the latest version of your application either automatically or as a one-click process.

In essence, Continuous Integration is about reducing risk by providing faster feedback. First and foremost, it is designed to help identify and fix integration and regression issues faster, resulting in smoother, quicker delivery, and fewer bugs. By providing better visibility for both technical and non-technical team members on the state of the project, Continuous Integration can open and facilitate communication channels between team members and encourage collaborative problem solving and process improvement. And, by automating the deployment process, Continuous Integration helps you get your software into the hands of the testers and the end users faster, more reliably, and with less effort.

This idea of automated deployment is important. Indeed, if you take automating the deployment process to its logical conclusion, you could push every build that passes the necessary automated tests into production. The practice of automatically deploying every successful build directly into production is generally known as Continuous Deployment.

However, a pure Continuous Deployment approach is not always appropriate for everyone. For example, many users would not appreciate new versions falling into their laps several times a week, and prefer a more predictable (and transparent) release cycle. Commercial and marketing considerations might also play a role in when a new release should actually be deployed.

The notion of Continuous Delivery is a slight variation on the idea of Continuous Deployment that takes into account these considerations. With Continuous Delivery, any and every successful build that has passed all the relevant automated tests and quality gates can potentially be deployed into production via a fully automated one-click process, and be in the hands of the end-user within minutes. However, the process is not automatic: it is the business, rather than IT, that decides the best time to deliver the latest changes.

So Continuous Integration techniques, and in particular Continuous Deployment and Continuous Delivery, are very much about providing value to the end user faster. How long does it take your team to get a small code change out to production? How much of this process involves problems that could have been fixed earlier, had you known about the code changes that Joe down the corridor was making? How much is taken up by labor-intensive manual testing by QA teams? How much involves manual deployment steps, the secrets of which are known only to a select few? CI is not a silver bullet by any means, but it can certainly help streamline many of these problems.

But Continuous Integration is a mindset as much as a toolset. To get the most out of CI, a team needs to adopt a CI mentality. For example, your projects must have a reliable, repeatable, and automated build process, involving no human intervention. Fixing broken builds should take an absolute priority, and not be left to stagnate. The deployment process should be automated, with no manual steps involved. And since the trust you place in your CI server depends to a great extent on the quality of your tests, the team needs to place a very strong emphasis on high quality tests and testing practices.

In this book we will be looking at how to implement a robust and comprehensive Continuous Integration solution using Jenkins or Hudson.

Introducing Jenkins (née Hudson)

Jenkins, originally called Hudson, is an open source Continuous Integration tool written in Java. Boasting a dominant market share, Jenkins is used by teams of all sizes, for projects in a wide variety of languages and technologies, including .NET, Ruby, Groovy, Grails, PHP and more, as well as Java. So what has made Jenkins such a success? And why use Jenkins for your CI infrastructure?

Firstly, Jenkins is easy to use. The user interface is simple, intuitive, and visually appealing, and Jenkins as a whole has a very low learning curve. As we will see in the next chapter, you can get started with Jenkins in a matter of minutes.

However Jenkins does not sacrifice power or extensibility: it is also extremely flexible and easy to adapt to your own purposes. Hundreds of open source plugins are available, with more coming out every week. These plugins cover everything from version control systems, build tools, code quality metrics, build notifiers, integration with external systems, UI customization, games, and much more. And installing them is quick and easy.

Last, but certainly not least, much of Jenkins’s popularity comes from the size and vibrancy of its community. The Jenkins community is a large, dynamic, reactive and welcoming bunch, with passionate champions, active mailing lists, IRC channels and a very vocal blog and twitter account. The development pace is fast, with releases coming out weekly with the latest new features, bug fixes, and plugin updates.

However Jenkins also caters to users who are not comfortable with upgrading on a weekly basis. For those who prefer a less-hectic release pace, there is also a Long-term Support, or LTS, release line that lags behind the latest release in favor of more stability and a slower rate of change. New LTS releases come out every three months or so, with important bug fixes being backported. This concept is similar to the Ubuntu LTS releases.

From Hudson to Jenkins—A Short History

Jenkins is the result of one visionary developer, Kohsuke Kawaguchi, who started the project as a hobby project under the name of Hudson in late 2004 whilst working at Sun. As Hudson evolved over the years, it was adopted by more and more teams within Sun for their own projects. By early 2008, Sun recognized the quality and value of the tool, and ask Kohsuke to work on Hudson full-time, starting to provide professional services and support around Hudson. By 2010, Hudson had become the leading Continuous Integration solution with a market share of over 70%.

In 2009, Oracle purchased Sun. Towards the end of 2010, tensions arose between the Hudson developer community and Oracle, initially triggered by problems with the Java.net infrastructure, and aggravated by issues related to Oracle’s claim to the Hudson trademark. These tensions also reflected strong underlying disagreements about the way the project was being managed by Oracle. Indeed, Oracle wanted to move towards a more strictly controlled development process with a slower release schedule, whereas most of the core Hudson developers, led by Kohsuke, preferred to continue with the open, flexible, and fast-paced community-focused model that had worked so well for Hudson in the past.

In January 2011, the Hudson developer community decisively voted to rename the project to Jenkins. They subsequently migrated the original Hudson code base to a new GitHub project and continued their work there. The vast majority of core and plugin developers upped camp and followed Kohsuke Kawaguchi and other core contributors to the Jenkins camp, where the bulk of the development activity can be seen today.

After the fork, a majority of users also followed the Jenkins developer community and switched to Jenkins. At the time of writing, polls show that some 75% of Hudson users had switched to Jenkins, while 13% were still using Hudson, and another 12% were using both Hudson and Jenkins or in the process of migrating to Jenkins.

Nevertheless, Oracle and Sonatype (the company behind Maven and Nexus) have continued to work on the Hudson code base (now also hosted on GitHub at https://github.com/hudson), but with a very different focus. Indeed, the Sonatype developers have concentrating on major underlying infrastructure changes around, among other areas, Maven integration, the dependency injection framework and the plugin architecture.

Should I Use Jenkins or Hudson?

So should you use Jenkins or Hudson? Since this is a book on Jenkins, here are a few reasons why you might want to opt for Jenkins:

• Jenkins is the new Hudson. In fact, Jenkins is simply the old Hudson with a new name, so if you liked Hudson, you’ll like Jenkins! Jenkins uses the Hudson code base, and the development team and project philosophy remain the same. In a nutshell, the original developers, who wrote the vast majority of the Hudson core, simply resumed business as usual after the fork working on the Jenkins project.

• The Jenkins community. Like many of the more successful Open Source projects, much of Hudson’s strength came from its large and dynamic community, and its massive adoption. Bugs are identified (and generally fixed) much more rapidly, and, if you have a problem, chances are someone else will have had it too! If you run into trouble, post a question on the mailing list or IRC channel—there’s sure to be someone who can help.

• The fast development pace. Jenkins continues the rapid release cycles that typified Hudson, which many developers love. New features, new plugins and bug fixes come out weekly, and the turn-around time for bug fixes can be very short indeed. And, if you prefer more stability, there are always the LTS releases

And, in the interest of balance, here are some reasons you might prefer to stick with Hudson:

• If it ain’t broke, don’t fix it. You already have a Hudson installation that you are happy with, and don’t feel the need to upgrade to the latest version.

• Enterprise integration and Sonatype tools. Hudson is likely to place a strong emphasis on integration with enterprise tools such as LDAP/Active Directory, and the Sonatype products such as Maven 3, Nexus and M2Eclipse, whereas Jenkins is more open to other competing tools such as Artifactory and Gradle.

• Plugin architecture. If you intend to write your own Jenkins/Hudson plugins, you should be aware that Sonatype is working on providing JSR-330 dependency injection for Hudson plugins. New developers may find this approach easier to use, though it does raise issues about future plugin compatibility between Jenkins and Hudson.

The good news is, no matter whether you are using Jenkins or Hudson, the products remain very similar, and the vast majority of techniques and tips discussed in this book will apply equally well to both. Indeed, to illustrate this point, many screenshots in this book refer to Hudson rather than Jenkins.

Introducing Continuous Integration into Your Organization

Continuous Integration is not an all-or-nothing affair. In fact, introducing CI into an organization takes you on a path that progresses through several distinct phases. Each of these phases involves incremental improvements to the technical infrastructure as well as, perhaps more importantly, improvements in the practices and culture of the development team itself. In the following paragraphs, I have tried to paint an approximate picture of each phase.

Phase 1—No Build Server

Initially, the team has no central build server of any kind. Software is built manually on a developer’s machine, though it may use an Ant script or similar to do so. Source code may be stored in a central source code repository, but developers do not necessarily commit their changes on a regular basis. Some time before a release is scheduled, a developer manually integrates the changes, a process which is generally associated with pain and suffering.

Phase 2—Nightly Builds

In this phase, the team has a build server, and automated builds are scheduled on a regular (typically nightly) basis. This build simply compiles the code, as there are no reliable or repeatable unit tests. Indeed, automated tests, if they are written, are not a mandatory part of the build process, and may well not run correctly at all. However developers now commit their changes regularly, at least at the end of every day. If a developer commits code changes that conflict with another developer’s work, the build server alerts the team via email the following morning. Nevertheless, the team still tends to use the build server for information purposes only—they feel little obligation to fix a broken build immediately, and builds may stay broken on the build server for some time.

Phase 3—Nightly Builds and Basic Automated Tests

The team is now starting to take Continuous Integration and automated testing more seriously. The build server is configured to kick off a build whenever new code is committed to the version control system, and team members are able to easily see what changes in the source code triggered a particular build, and what issues these changes address. In addition, the build script compiles the application and runs a set of automated unit and/or integration tests. In addition to email, the build server also alerts team members of integration issues using more proactive channels such as Instant Messaging. Broken builds are now generally fixed quickly.

Phase 4—Enter the Metrics

Automated code quality and code coverage metrics are now run to help evaluate the quality of the code base and (to some extent, at least) the relevance and effectiveness of the tests. The code quality build also automatically generates API documentation for the application. All this helps teams keep the quality of the code base high, alerting team members if good testing practices are slipping. The team has also set up a “build radiator,” a dashboard view of the project status that is displayed on a prominent screen visible to all team members.

Phase 5—Getting More Serious About Testing

The benefits of Continuous Integration are closely related to solid testing practices. Now, practices like Test-Driven Development are more widely practiced, resulting in a growing confidence in the results of the automated builds. The application is no longer simply compiled and tested, but if the tests pass, it is automatically deployed to an application server for more comprehensive end-to-end tests and performance tests.

Phase 6—Automated Acceptance Tests and More Automated Deployment

Acceptance-Test Driven Development is practiced, guiding development efforts and providing high-level reporting on the state of the project. These automated tests use Behavior-Driven Development and Acceptance-Test Driven Development tools to act as communication and documentation tools and documentation as much as testing tools, publishing reports on test results in business terms that non-developers can understand. Since these high-level tests are automated at an early stage in the development process, they also provide a clear idea of what features have been implemented, and which remain to be done. The application is automatically deployed into test environments for testing by the QA team either as changes are committed, or on a nightly basis; a version can be deployed (or “promoted”) to UAT and possibly production environments using a manually-triggered build when testers consider it ready. The team is also capable of using the build server to back out a release, rolling back to a previous release, if something goes horribly wrong.

Phase 7—Continuous Deployment

Confidence in the automated unit, integration and acceptance tests is now such that teams can apply the automated deployment techniques developed in the previous phase to push out new changes directly into production.

The progression between levels here is of course somewhat approximate, and may not always match real-world situations. For example, you may well introduce automated web tests before integrating code quality and code coverage reporting. However, it should give a general idea of how implementing a Continuous Integration strategy in a real world organization generally works.

Where to Now?

Throughout the remainder of this book, as we study the various features Jenkins has to offer, as well as the practices required to make the most of these features, we will see how we can progress through each of these levels with Jenkins. And remember, most of the examples used in the book are available online (see http://www.wakaleo.com/books/jenkins-the-definitive-guide for more details), so you can get your hands dirty too!