As part of the GDS commitment to code in the open, we have opened up GOV.UK’s application deployment code. This came with a number of challenges but it increased the development team’s efficiency and collaboration. Find out why we opened the code and how we did it.
Why we opened the code
Since the early days of GOV.UK, our Capistrano-based scripts for deploying applications were kept in a private repository called alphagov-deployment. This repo also contained lots of environment-specific configuration and sensitive information needed by our applications, such as database passwords.
Because the repo contained production secrets, we had to limit access to specific developers who had spent time on our technical support team. This made the simplest changes difficult. Whenever developers without access were working on something that needed a change in how an application was deployed, they had to find a colleague with the right permissions to help them. This workflow caused avoidable delays for teams and made it harder for those without access to understand how deployments worked.
The lack of separation between code, configuration and secrets also went against the principles of twelve-factor apps, which we try to follow at GDS. We’d been intending to move all application configuration into environment variables (the third of the twelve factors), and we’d started setting up new applications in that way. Moving the existing applications to use environment variables was time-consuming and fiddly to do, so we’d made little progress on them.
We decided to tackle this problem when 2 of our junior developers started working on a new application as part of our local links mission. They needed to set up asset compilation in its deployment scripts but couldn’t because they didn’t have the necessary access privileges - even though the deployment code itself didn’t need to be kept private and all the credentials needed by the new application were already in environment variables.
The development team decided to move the deployment code from the private repository to somewhere accessible for all the developers working on it.
Creating a new repo and splitting the code
Originally we thought we'd put the deployment code in each individual app's repository - here’s an early idea of what we were thinking.
We decided against doing this because we wanted to avoid duplicating all of the generic code in each repo. Instead we moved all the deployment scripts to a new public repo on GitHub called alphagov/govuk-app-deployment. This allowed us to keep the code in one place, which made it easier to manage and understand the differences between applications.
Once that was set up and we’d added support for deploying using the new repo, it was a fairly simple task to move the deployment code for each application out of the old repo and into the new one. The secrets were already in separate files from the deployment code, so moving across the right stuff was straightforward.
Our team mobbed on the early stages of the work. This helped our junior developers learn how all the parts of our infrastructure fit together and change how all of GOV.UK’s applications are deployed, all within their first month at GDS.
We decided it wasn’t worth trying to clean up the history of the repo to preserve that in public. That would be time-consuming and we were confident that the history wasn’t particularly valuable for this code (we still have this history in the old private repo).
Making the decision to move the code out of the old repo meant we could make it public without having to invest the time in moving everything to environment variables right away (configuration was left in the old repo for the time being). Once we’d finished this piece of work we could deprecate alphagov-deployment and make it clearer that it shouldn’t be used for any new configuration. We were also able to see more clearly what secrets were left in the old repo and plan how to deal with them.
Modifying the deployment code
With security in mind, we made the new repo private to start with just in case anything confidential was accidentally pushed to it. Once we’d moved the first few apps across safely, we were confident that the process could be applied to all apps. We made the new repo public within a couple of weeks of starting the work and continued the process in the open.
We continued to support our old way of getting files containing secrets and environment-specific configuration into place on production servers. That meant we avoided having to do the work of switching to use environment variables for all our configuration up front.
The deployment code for some apps copies files from alphagov-deployment into place after putting the new version of the code onto the servers, for example it may overwrite the database.yml file from the application repo with one from alphagov-deployment containing the production database credentials. We needed to modify the deployment code to look for those files to copy in an application-specific directory called secrets so that this legacy mechanism would continue to work.
This was the only change we needed to make to the deployment code as we moved each app across to use the new repo. We’ll be able to get rid of this complexity in the deployment process once we’ve moved to environment variables for all our configuration.
We had a lot of support for the project across GOV.UK. Everyone we spoke to recognised this was something that was holding up work and needed to be fixed. However, it took some time for every team to move their code across - GOV.UK deploys about 70 applications owned by 8 product teams.
Our team transferred our apps within a few weeks but a couple of other teams weren’t able to prioritise the work for several months. The early days of the project were in April 2016, and the vast majority of applications were moved by December 2016. All the teams are also making good progress with moving to use environment variables for our config.
As well as the deployment code, the public repo also contains associated code, such as for announcing deploys in Slack.
Open is better
I'm really pleased this code is now open. If you want to build your own GOV.UK, you can use this code to do that - it’s the missing piece between our infrastructure configuration (which is also open) and our application code, along with our developer documentation.
It's great that two of our junior technologists were involved with this so early in their GDS careers. They've learned a lot about how our infrastructure works in the process, and we’ve had the opportunity to reinforce an important aspect of GDS culture - it’s ok to go ahead and change something that’s getting in the way of your work.
Having this code in the open makes it clearer how GOV.UK works and makes working on it easier for all our developers. As our 10th design principle says - Make things open: it makes things better.
Stay up to date with all the latest posts by signing up to alerts from Government Technology blog and joining the cross-government conversation on Slack.
1 comment
Comment by Bob posted on
Why you shouldn't use environment variables for secrets - https://diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/