Continuous Delivery: The Henry Ford approach to software development

The assembly line was invented over 1,000 years ago in China. State-run workshops used the idea of division of labour to allow people to work more efficiently by removing the need to switch from one task to another. In this way they were able to mass produce weapons, pottery and farming tools. The Venetians used similar ideas when they created the Venetian Arsenal in 1104. At the Springfield Armory in the early 19th century, interchangeable components were created for the manufacture of weapons. This allowed the stages in the assembly line to be more loosely coupled because components no longer had to be crafted to fit into other specific components.

And then came Henry Ford. Ford is often credited as the inventor of the modern assembly line. His particular innovation was the introduction of continuous processes. In the Ford plant specialists created components in a long connected process in the same way that they had for over 1,000 years. But Ford connected those separate stages with conveyor belts. No longer would workers created a batch of widgets and store them for the next stage in the process. Instead a long continuous stream of components was manufactured and assembled. Production stages were broken down until they ran at the speed of the conveyor belt. Efficiency rose and production costs dropped dramatically.

Since Ford, continuous processes have been to all manner of business: from cake manufacture to package delivery. And, most importantly for a company like Black Pepper, in software production.

In Extreme Programming Explained, Kent Beck identified the principle of Flow for any successful software project. Flow means, quite simply, that processes should be as continuous as possible. When two people pair program, that's an example of continuous code review. Story boards are an example of continuous project management. And increasingly software is delivered continuously.

Continuous delivery versus continuous deployment

Continuous delivery means that software is continuously built and deployed onto some working platform. It doesn't mean that the code is necessarily deployed to production. If you are continuously deploying new versions of code to production, you are doing continuous deployment. Sites like Flickr are famous for continuously deploying new versions of code into live. Most small to medium businesses opt instead for continuous delivery, because it will allow them to decide when new features go live.

In either case, an automated assembly line is required to convert the latest code changes into code that has been built, tested and deployed. If it is working efficiently, this assembly line can more rapidly feedback errors in the latest code and massively reduce the cost of deployment. If you are involved in software in any way, you should consider it as a way of of reducing costs and improving overall code quality.

What do you need to do to introduce continuous delivery or deployment? Here are 10 example steps.

1. Identify the latest code

This might sound like an obvious first step. If you are not able to point to a complete set of source code and say "This is the current version", you will not be able to introduce a continuous deployment assembly line.

This means if you have your code base spread across several repositories, you are likely to have huge problems. If, say, you store your server code in a different place from your client code, you will likely have compatibility issues whenever someone makes an amendment to the server and the client. You can reduce the likelihood of this by having formal interface definitions. You could, say, define a very strict server API with a tool like Swagger, and treat your server code as a kind of in-house third party service. You can then develop your various components in a more de-coupled fashion. Each of those components can then have their own, separate, continuous delivery pipelines.

For most businesses, the simplest thing is to put all of your code in a single repository. That will allow you to say, at any single moment, this is our latest code.

2. Remove human steps from your build

If you need to copy files, or update version numbers, or send an email or interact with a GUI at any point between writing code and creating the runnable artifacts, then you will need to change the way your build works. You need to have a build that can be started by a simple, single command. This used to be a bigger problem than it now is. Historically it was quite difficult to automate a build that required an interaction with an IDE. These days, most IDEs use command line build systems like grunt, gulp or gradle to do the heavy lifting.

The important thing about having a simple build command is that it makes it possible to…

3. Automate your build

Humans are expensive; computers are cheap. You should not need a human being to repeatedly check out the latest code and build it. Instead, give the job to a computer. Build a continuous integration server, or sign up for a hosted service.

For example, you might want to run a system like Jenkins or subscribe to a service like Gitlab.

If you work with code branches, consider allowing your integration server to build every code branch that is sent back to the source code repository. This might seem initially wasteful, because many teams will have as many branches as there are coding pairs. But don't worry about it. You will likely need a more powerful server to build each branch, but the cost of that additional power will be far lower then the cost of the additional developer time if you don't.

4. Automate your unit tests

OK, hopefully you are already doing this. If not, why not? Creating automated tests does take longer than simply writing the code and manually testing it… the first time. But as you want a feature to be part of your code, you will need to test if after every single change to the system, just to make sure that some distant corner of the code isn't doing something to break the feature you just added.

5. Automate code quality checks

OK, this one is a little more controversial. Many developers have worked on systems where automated code quality checks have led to localised insanity in the code. Test coverage checks that check the code was run, but not that it necessarily changed anything. Function size checks that cause developers to remove all blank lines from their code, and so on.

It's true that automated code quality checks do not, currently, have anywhere near enough intelligence to replace human beings who understand what code should do and why it should do it. But they reduce some of the heavy lifting. For example, code formatting is the kind of thing that can lead to endless arguments between developers. A tool can simply check if the code matches formatting standards. It doesn't matter what those standards are, just so long as everyone agrees with them. Test coverage checks can highlight which areas of the code would benefit from more testing. Complexity checks will help you track down gnarly pieces of complex code that could be refactored to make them simpler.

Consider using a tool like Sonar, which allows you to flag which code checks you disagree with. You may choose to use simply report issues from an automated quality check, rather than breaking the entire build. But the important thing is to treat it as a money saver. It will save you hours of arguing about stuff that doesn't matter so much.

6. Archive your build artifacts

Assuming that your code has got through all of the huddles so far, you should consider creating an archived (timestamped and versioned) copy of the built artifacts. You don't need to take the risk of re-building the code, just in case external dependencies, just as third-party libraries, change. For the rest of the pipeline, work with these built artifacts. That instantly means that you should not have hard-coded environment linkages in your code: for things like databases and web services. Those things should be part of your deployment configuration.

Talking of which…

7. Automate your deployment

Automate the process of deploying your code somewhere. Not necessarily to a live environment. Allow the deployment to be configurable. Jez Humble and Dave Farley (the authors of the book Continuous Delivery) wrote their book from their experience of creating an automated deployment tool called Conan the Deployer. It was a bash script. Go and do likewise. Create a single command that will completely deploy everything needed by your system. I once wrote a system that deployed a complete Point of Sale system across multiple tills, databases and application servers from a Python script.

It might take some time to automate the deployment, but it will reap a rich harvest the next time you need to deploy a new version of the system to a live environment. If you do it correctly, and if you use the same basic process for any environment you deploy to, then you should greatly reduce the need for weekend working and late night call-outs the next time you send a new version of the code live.

8. Automate your acceptance tests

It costs a lot for a tester to run through a deployed version of a system, clicking buttons and checking that the lights are working. As the number of features in a system increases, it will also became an increasingly arduous burden to re-test all of the old features. Over time it will eventually become impossible to test all the old features. The testers simply won't have time. So you should automate the UI tests as much as possible. You probably won't be able to automate everything, because it's hard to write a test that checks, for example, visual stuff like alignments. But if you automate the bulk of the tests, your test team will have far more time to check if a feature is genuinely appropriate, and to help define automated tests for future changes.

Consider using a framework like Cucumber to test your UI. It will allow you to write tests that read like business requirements. Writing business-level tests will greatly reduce the likelihood of errors being discovered when a system is deployed to an acceptance testing environment.

Ooh, that reminds me.

9. Automate a deployment to an Acceptance Test environment

Everything so far has been appropriate to both continuous delivery and continuous deployment. But continuous deployment to a production system has a much more aggressive pipeline from this point. It really requires there to be no human interactions with the process. Your code checks and your tests should be sufficiently strong that any code that gets through them is instantly good enough to go to the production environment.

However, if you are doing continuous delivery, you can allow yourself a little human involvement. Many businesses benefit from deploying to a User Acceptance Testing (UAT) environment. This allows people from the business to check through the latest version of the code and have the ability to decide if the code is in a fit state to go live. Quite often this will merely be a comfort check. If you find that your builds are often rejected at the UAT stage, consider improving your automated acceptance tests.

If you get through UAT, congratulations; your code can go live. Well it might.

10. Automate a deployment to non-functional testing environments

There is the little matter of non-functional testing: stuff like security penetration tests and performance. Some security checks can be automated, but they often rely on external specialists who will take some time performing an audit. Generally, performance testing can be automated, but it will often take a long time to run. For this reason, non-functional tests are carried out in parallel environments. They might take place at the same time as UAT, or (in the case of penetration testing) they might take place several times a year in the form of a general audit.

It's a business decision as to whether non-functional tests are on the critical path to production deployment.

Other considerations

Some businesses will choose to adopt other development practices when working in a continuous delivery environment. The most controversial proposal from Humble and Farley's Continuous Delivery is probably doing away with code branches. A code branch allows development to take place in a parallel world from current live production code. You might want to ban code branches if you want to encourage developers to make smaller check-ins of completed code. This will get value out to the customer more rapidly, and it will also reduce the amount of time spent resolving conflicts between branches. Opponents to this idea would point out banning branches might simply encourage developers to check in less frequently, if they feel that their code is not yet production ready. And tools like git allow branch conflicts to be resolved in a more continuous way using frequent rebases on production code. What should you do? Probably read the book and make a conscious decision.

If you are doing continuous deployment, you might want to consider using feature flags. A feature flag is a configuration setting which says whether a new feature is available. Some people inherently dislike feature flags because they can be used to add more code complexity before it is genuinely needed. For example, a large set of features might be checked in to the code, and disabled by a flag, in the hope that at some point in the near future this set of features will be classed as "complete" and can then be enabled. Or you may discover that at some future point that they are no longer needed and cannot be easily removed from the code. However, one very great advantage of feature flags is that they allow you to enable new code for some users and experiment to see if they are genuinely popular. This is primarily important for applications that have a very large number of users: the very systems that most benefit from continuous deployment.

In Summary

Continuous processes make your business more efficient. Whether you are making cars on a production line, or baking cakes on a conveyor belt, continuous processes are far less wasteful than processes which stop and start. And continuous delivery/deployment is a great way of lowering the cost of the important, but often tedious, process of getting newly crafted software into the hands of people who will find it valuable. If you don't currently have an automated system for this, you are most likely spending money where you don't need to. At the same time, it will likely take some time to automate as much as possible. The good news is that you can, as stated above, move to a continuous pipeline in a series of stages. The early stages (identifying the latest code, automated build, automated unit tests) are the easiest and cheapest to implement.

Each of these steps will take effort, but for each stage, it is worth bearing in mind that this will save you money.