Continuous Delivery: The Good, the Bad, and the Ugly
I was recently invited to speak at UXDX in Dublin on the topic of continuous delivery and how it impacts product delivery
At Dashlane we have invested heavily in improving our engineering practices, aiming for painless, fast, and reliable releases. This is an ongoing investment with no target destination, focused on continuously getting better. Here are a few learnings we can share so far, ranging from huge successes to some of the challenges we have faced.
The Good
In the summer 2018, we unveiled Dashlane 6: a massive release packed with new stuff for our users, supporting an important marketing campaign. On that day, we pressed the button in our back office, and almost instantaneously cool new features became available to millions of Dashlane users. From an engineering perspective, there was no rush hour, no scrambling to make sure everything was ready at the last minute. Code had been shipped progressively in the past weeks and months, hidden behind feature flips. We had pre-tested with our beta users and A/B tested where needed ahead of time, so we could have a smooth ride on D-Day.
Continuous delivery has benefits across the company, and relieves much of the stress that normally comes with new product launches or substantial marketing campaigns.
What is Continuous Delivery?
My own definition is: an engineering practice that aims at enabling the ability to quickly and reliably deliver to production on-demand, through an industrialized, automated release pipeline.
Continuous delivery is costly, but worth the time and money. The primary costs come from the following three prerequisites:
- High-performing code practices
Your code must be centralized in a single code repository, orchestrated thanks to a strong continuous integration platform (this is the core of your engineering system). It is important to support code quality through automated tools and systematic mandatory code reviews. The key metric here is how fast can a line of code move through your pipeline, meaning you need to aim for short build times, efficient notification mechanisms, and high reaction time from the team. A healthy code review practice has a lot of implications for an engineering group, its efficiency and its morale. If developers don’t align on the expectations between them, you generate a lot of waiting time. One developer will commit some code and then be blocked by their peers as they review. We learned the hard way that it is key for developers to submit small amounts of code that can be reviewed quickly, to keep a smooth flow. Code review should be one of the top priority activities for a developer. - Continuous QA
Ensure you have full-stack testing: unit testing of the code, integration testing of your various components and modules, as well as functional testing. This is called the pyramid of tests. The base, unit testing should be fully automated with high code coverage, and then as you move up the pyramid, you should automate as much as possible, while balancing with the cost of maintaining stable test scenarios. Test automation is a heavy investment, but comes with significant ROI in the form of quality and stability. When I joined Dashlane, we had limited and inconsistent practices. We had to train both our development and QA teams. We hired dedicated test automation engineers to build and maintain the test frameworks. We are way better today, but this keeps being an area of effort. - Continuous operations
When shipping to production, you must set up continuous operations, based on a monitoring platform that is as exhaustive as possible. It must include technical, product, and business metrics, as well as an alerting system that will warn you in advance before your users realize a problem is happening. It is not as easy as it seems. If you don’t choose the right thresholds you will generate a lot of false positives, and the team will start ignoring alerts. A good practice that we have is what we call “Morning Gymnastics”. In each team, someone is in charge of proactively checking dashboards and metrics to anticipate issues and warn the rest of the team. The goal here is to fix issues before they hit our customer support.
Now this will only give you a pipe, where code flows. You need to open the tap at the end.
Progressive rollout is what you need to deliver code safely to production. Most tech platforms support progressive rollout today, so make use of it. You can open the tap slowly, shipping your code to a small percentage of your user base, monitoring if things are okay before deploying to all.
On a recent iOS release, code was being rolled out to 1% of our users and we identified a spike in crashes. This had gone unnoticed through QA because it was related to a production certificate generated by Apple. We stopped the deployment, found the source, hotfixed, and were able to resume our rollout, confident that the crashes had impacted a reduced number of users and didn't overwhelm our customer support team.
Another key tool of continuous delivery is feature flipping. Feature flipping gives you the ability to control the activation of a feature from the back-office for a certain segment of users. At Dashlane, this is owned by our product managers. They define the type of feature flips they want to use for each new feature or experimentation, and are in charge of deciding when the moment is right to activate a new feature to which group of users.
One challenge with feature flipping is the small technical debt it generates in the code – it is important to clean up the feature flipping code after the fact to avoid clutter. Feature flips need to remain a deployment tool, and should not be used for other means. We had teams that started using it as a way to configure the application for specific populations and for A/B testing. Don’t make the same mistake. Instead, create a real configuration mechanism for your application, and use a real A/B test engine.
On the topic of A/B test engines, they are necessary to experiment in production. But if you have many teams running A/B tests in parallel, make sure they communicate together and synchronize. Ideally they should be testing clearly separate parts of your app to avoid biases.
With every tool, it is how you use them that will matter and make an impact on the business.
The Bad
As you accelerate, it is critical that the whole team is synchronized—from engineering down to marketing—and aware of the side-effects associated with continuous delivery practices. Of course, it gives you a lot of control and flexibility, but it means you need to be even more strict on following what’s happening between progressive rollout, activation of feature flips or running A/B tests. Earlier this year, we released an update to our VPN feature so that Dashlane users could select a specific country. The feature had gone smoothly through phases, and the marketing team launched campaigns to announce it. Then we started getting hundreds of tickets in customer support from users complaining that the feature was not available to them. It appeared that the macOS release had been stuck in Apple validation and never reached production.
Know your release life cycle and communicate. It is the team’s responsibility to track which version is in production on each platform and what feature has been activated for which users. It does not stop with shipping the code.
The Ugly
Another challenge we have encountered is related to output vs outcome. Continuous delivery will give you the ability to have intense throughput of delivery, but this is just an enabler. By investing in continuous delivery, you are just reducing a bottleneck. What you put in the pipe is what is important above all. Make sure your product & engineering organization maximizes for impact and outcome not for output. That technical capability and those tools are necessary but not sufficient to make an impact on the business.
If you put crap in, you will get crap out.
Four Final Takeaways
- Investing in continuous delivery practices will give you the ability to accelerate your business.
- It requires deep engineering practices around code, QA, and operations, as well as appropriate tooling: progressive rollout, feature flipping, and an A/B test engine.
- As you accelerate, make sure you double down on communication and synchronization between teams.
- Continuous delivery is just about the plumbing. Make sure you have liquid gold flowing through it to make a real impact.
What about you? What anecdote can you share about your delivery practices? Feel free to share in the comments.
If you want to contribute to improving the way we do things at Dashlane, we are hiring.
Sign up to receive news and updates about Dashlane
