Building a New A/B Testing Engine at Dashlane

Let's say you are updating some feature of your product, or adding a new one. You want to make sure it'll have a positive impact.

There are several ways to do that: "Before / After" method is the most simple one. You release and then monitor results. Eventually, the results are positive and you maintain your course, or the results are underwhelming and you roll back the changes. But sometimes, you don't want to take the risk and you need a way to measure the impact before you release your change to everyone.

A/B testing is one solution to that problem. At a high level, it consists of following steps:

In this post, I'll discuss how we think about A/B testing at Dashlane, and why we recently invested in developing a new testing engine and framework in-house instead of purchasing a third party solution.

A/B testing is one of Dashlane's key growth levers. Each day, we have dozens of A/B tests running to identify the optimal user experience for our customers. We test a range of experiences, from the order we show customers our features to onboarding, in-app design, messaging, and more.

For example, at some point we wanted to improve the account creation flow with a new design.

We were so confident that the experiment experience was superior, we actually considered a full rollout. But instead of trusting our intuition, we ran an A/B Test to get a quantifiable result. Much to our disappointment, we saw a 16% drop in the metric we were tracking. Our intuitions were totally wrong, and without A/B testing we only would have witnessed this decline in performance after implementing to 100% of our users, rather than a smaller, randomized cohort.

Because A/B testing minimizes the risk of a new experience hurting your core metrics, it also unlocks a greater willingness to try lots of new things and see what works. Recently, as part of an effort to improve iOS account creation rates, we wanted to test whether external validation of Dashlane would increase customers' willingness to give our product a try. We tried to adopt an idea that is becoming industry best practice: highlighting some of the awards and recognition we received over the years. Our thinking was that this would give prospective customers more confidence that Dashlane was a safe and effective product. We figured external validation from recognizable brands and thousands of happy customers might do the trick.

Around 50k users were involved in the test, and in short order the experiment revealed a 4% increase in account creation compared to our control group. So we decided to rollout this slideshow to every new user!

Until recently, Dashlane used to conduct tests with what we now call "The Old Engine." Built few years ago, its initial goal was to test pricing and plans. Dashlane was a much smaller company at the time – operating on fewer platforms, and too few users to run multiple tests in parallel and reach statistical significance in a reasonable amount of time. The old engine became increasingly unreliable as we grew in size, and as we identified mistakes in our initial framework for A/B testing , which sometimes led to unreliable results or non-optimal user exposure. These mistakes included:

When we reached consensus to change our A/B testing engine, the first question was: "should we do it ourselves, or should we rely on a third party / open source one instead?"

We saw a few issues with using a third party platform:

So we decided to do it on our own and benefit from all the cumulative knowledge and experiences from our Old Engine. Our main upfront constraints were the ability to conduct several tests per user (so that we could run more tests in parallel), and ensuring the reliability of our testing framework so we were confident in the results. To face this challenge, we needed two really important things: a code name and a motto.

Because we have a lot of imagination here on the Server team, we named it... "The New A/B Testing Engine."

The MOTTO was a bit more scientifically chosen by our Data Analyst involved in the project - "Let's Compare Bananas with Bananas!"

To avoid selection biases, we built our new engine with several precautions in mind:

To maximize the number of tests we can run in parallel, we reused parts of our Feature Flip engine (our tool for controlling progressive roll-outs) that make targeting specific segments a more flexible and customizable process.

All this work means:

That's it. Thanks for your time reading, and if you have any share-worthy learnings on A/B testing, please share in our comments section!

And did we mention that we're hiring engineers in New York, Paris, and Lisbon? Check out our careers page!

Julien Beaussier

Read more