We have just over 50 applications that we release to GOV.UK, some more regularly than others. For example, Whitehall is released every day, whereas our release app is ironically released about once every six months.
Since launch in October 2012 we have released applications several thousand times to our production environment. All of our deployments are tracked by our release app that shows which version of an application has been deployed to each of our environments.
Our release process
As our development teams are building their applications, our Continuous Integration system creates a tag on every successful build of an application and merges master into a release branch. A successful build triggers a deploy of the release branch to our preview environment so that the team can check their changes in a production-like environment and to smoke out any major errors.
When a team has a build they want to deploy to production they do the following:
- Book a deploy slot (normally 30 minutes) in the release calendar giving details of the application, tag and expected changes.
- Check any notes against the application in the release app.
- Acquire the Badger of Deploy from 2nd line.
- Ensure the Badger of Deploy is not tagged with a note about the application being deployed, indicating that it is blocked from being released.
- Deploy to staging using the Jenkins job.
- Check Smokey passes.
- Check the new functionality works as you would expect.
- Take a look at any alerts and metrics, just to check you haven’t broken something.
- Repeat steps 5-8 in production.
- Return the Badger of Deploy to someone on 2nd line.
- Stay around in the office for a while, just in case something goes wrong.
Our release principles
We have several principles that we work to when we do a release.
- Releases should be zero downtime
A release should not cause any downtime. For most of our apps this is achieved by using Unicorn, which starts up new processes which deal with new requests. The old requests are still processed by the previous release until they are finished. However, this means Upstart loses track of the process ID, so to solve this problem, we wrote Unicorn Herder which wraps Unicorn.
- Teams are responsible for deploying their own work
The team which wrote the code are in the best place to debug any issues which may occur during their deploy. Ownership of the entire process is important and encourages people to fix any pain points that they experience.
- One product team owns each release
We have several applications which are developed by several different teams. When releasing only one of them should own it. This is currently decided by whose code was merged to master first. This also encourages frequent releases so that teams are only releasing their own code.
- Deployments should be repeatable
The usual pattern for using Capistrano is that each host has a clone of the repository and will update that local copy on deploy. However, not all of our code is hosted publicly on GitHub; some is on internally hosted Git repositories for added security. To deal with this we use the rsync with remote cache deployment strategy for Capistrano so that only our Jenkins machine has to have a VPN connection to where our private repositories reside.
- Applications should be released regularly
Releasing regularly means that only small changes will be released which makes the job of debugging any issues after a deploy easier.
- Only one release at a time
Having only one thing going out to production helps to narrow the list of suspects if debugging is required. To ensure that only one release happens at a time we have The Badger of Deploy (a stuffed toy). The Badger of Deploy acts as a physical token that must be in the possession of anyone releasing an application.
- Normal releases should only happen during working hours
This is so should something go wrong with recently deployed code, a large number of the GOV.UK development team will be available to help with fixing it. Also it means that the person deploying is unlikely to go home leaving GOV.UK broken.
- Changes should be backwards compatible, except in exceptional circumstances
With approximately 50 applications if you make a breaking change you may end up having to deploy lots of applications which will make your life hard.
- Staging should be a maximum of one release ahead of production
We try to avoid releasing multiple applications at once to staging, preferring to release one application to staging, then production, before moving on to the next. This ensures that problems in any environment can be isolated to one application and one release, making them easier to debug.
If a release fails on staging, we roll back to the previous version unless we can fix it within the release slot.
- Deployments should be tested for regressions
With around 50 applications, a change to one may have effects elsewhere, therefore it is good to run tests after a deploy to check that the site is running as you would expect.
We achieve this by using Smokey which is a collection of behavior-driven development (BDD) tests written in Cucumber, to test the important functionality of GOV.UK. Different teams have contributed Smokey tests to make sure their applications are tested on every deploy.
Smokey is triggered to run after every deploy and is also continuously run and monitored by our monitoring stack.
Because we're government, we also have some additional constraints around who has access to production. In general, production access requires a track record at GDS and security clearance.
If this sounds like a good place to work, take a look at Working for GDS - we're usually in search of talented people to come and join the team.