Intro

You have a critical bug in production. It is losing money for your business every day. You know what the fix is: A one-liner in a library that is used in all three layers of your three-tier system, and a corresponding change to one database table. But the last time you released a new version of your software to production it took a weekend of working until 3 A.M., and the person who did the deployment quit in disgust shortly afterward. You know the next release is going to overrun the weekend, which means the application will be down for a period during the business week. If only the business understood our problems.

Done Means Released

How often have you heard a developer say a story or feature is “done”? Perhaps you have heard a project manager asking that developer if it is “done done”? What does “done” mean? Really, a feature is only done when it is delivering value to users. This is part of the motivation behind the practice of continuous deployment (see Chapter 10, “Deploying and Releasing Applications”).

For some agile delivery teams, “done” means released into production. This is the ideal situation for a software development project. However, it is not always practical to use this as a measure of done. The initial release of a software system can take a while before it is in a state where real external users are getting benefit from it. So we will dial back to the next best option and say that a functionality is “done” once it has been successfully showcased, that is, demonstrated to, and tried by, representatives of the user community, from a production-like environment.

There is no “80% done.” Things are either done, or they are not. It is possible to estimate the work remaining before something is done—but those will only ever be estimates. Using an estimate to determine the total amount of remaining work leads to recriminations and finger-pointing when those quoting the percentage turn out, as they invariably do, to be wrong.

How branching affects software delivery or why it's important to check in regularly in trunk

There is a tension at the heart of working with version control. On one hand, to gain access to many of its benefits, such as the ability to step back to a recent, known-good version of your artifacts, it is important to check in frequently.

On the other hand, once you check your changes into version control, they become public, instantly available to everybody else on the team. Further, if you are using continuous integration, as we recommend, your changes are not only visible to the other developers on the team; you have just given birth to a build that could potentially end up in acceptance testing or even production.

Since checking in is a form of publication, it is important to be sure that your work, whatever it may be, is ready for the level of publicity that a check-in implies. This applies to developers in particular who, given the nature of their work, need to be cautious about the effects of their check-ins. If a developer is in the middle of working on a complex part of the system, they won’t want to commit their code until it is finished; they want to feel confident that their code is in a good state and won’t adversely affect other functions of the system.

In some teams, this can lead to days or even weeks between check-ins, which is problematic. The benefits of version control are enhanced when you commit regularly. In particular, it is impossible to safely refactor an application unless everybody commits frequently to mainline—the merges become too complex. If you commit frequently, your changes are available for other people to see and interact with, you get a clear indication that your changes haven’t broken the application, and merges are always small and manageable.

A solution that some people use to resolve this dilemma is to create a separate branch within the version control system for new functionality. At some point, when the changes are deemed satisfactory, they will be merged into the main development branch. This is a bit like a two-stage check-in; in fact, some version control systems work naturally in this way.

However, we are opposed to this practice (with three exceptions, discussed in Chapter 14). This is a controversial viewpoint, especially to users of tools like ClearCase. There are a few problems with this approach.

• It is antithetical to continuous integration, since the creation of a branch defers the integration of new functionality, and integration problems are only found when the branch is merged.
• If several developers create branches, the problem increases exponentially, and the merge process can become absurdly complex.
• Although there are some great tools for automated merging, these don’t solve semantic conflicts, such as somebody renaming a method in one branch while somebody else adds a new call to that method in another branch.
• It becomes very hard to refactor the codebase, since branches tend to touch many files which makes merging even more difficult.

On importance of build CI server and build failure visibility

Projects use gadgets like these for the simple reason: They’re a great way to allow everyone to see the status of the build at a glance. Visibility is one of the most important benefits of using a CI server. Most CI server software ships with a widget that you can install on your development machine to show you the status of the build in the corner of your desktop. Tools like this are especially useful for teams that are distributed, or at least not working in the same room together.

The only drawback of such visibility is that if your development team is working in close quarters with their customers, as should be the case in most agile projects, build failures—a natural part of the process—may become regarded as a sign of problems with the quality of the application. The fact is that the reverse is true: Every time a build fails, it indicates that a problem has been found that may otherwise have made it into production. However, this can sometimes be hard to explain. Having been through this several times, including having some difficult conversations with clients when the build was broken for a longer period than any of us liked, we can only recommend that you keep the high-visibility build monitor and work hard at explaining its very real benefits. Of course, the best answer of all is to work hard to keep the build green.

Alternatives to continuous integration using DVCS

At one more step away from continuous integration is what Martin Fowler calls “promiscuous integration”. In this model, contributors pull changes not just between forks and the central repository, but also between forks. This pattern is common in larger projects that use GitHub, when some developers are working on what are effectively long-lived feature branches and pull changes from other repositories that are forked off the feature branch. Indeed in this model there need not even be one privileged repository. A particular release of the software could come from any of the forks, provided it passed all the tests and was accepted by the project leaders. This model takes the possibilities of DVCS to their logical conclusion.

These alternatives to continuous integration can create high-quality, working software. However, this is only possible under the following conditions:

• A small and very experienced team of committers who manage pulling patches, tend the automated tests, and ensure the quality of the software.
• Regular pulling from forks, so as to avoid large amounts of hard-to-merge inventory accumulating on them. This condition is especially important if there is a strict release schedule, because the temptation is to leave merging till near the release, at which point it becomes extremely painful—the exact problem that continuous integration is designed to solve.
• A relatively small set of core developers, perhaps supplemented by a larger community which contributes at a relatively slow pace. This is what makes the merges tractable.

These conditions hold for most open source projects, and for small teams in general. However, they very rarely hold for medium or large teams of full-time developers.

To summarize: In general, distributed version control systems are a great advance and provide powerful tools for collaboration, whether or not you are working on a distributed project. DVCSs can be extremely effective as part of a traditional continuous integration system, in which there is a designated central repository to which everybody regularly pushes their changes (at least once a day). They can also be used in other patterns that do not allow for continuous integration, but may still be effective patterns for delivering software. However, we caution against using these patterns when the right conditions, listed above, are not satisfied.