How Dashlane Upgraded Its Back-End Alerts System

When Dashlane’s User Support team contacted me to tell me that our users couldn’t create accounts, I felt like an idiot.

I am, after all, in charge of Dashlane’s back-end systems.

But when I looked at the monitoring we set up, I saw that all our instances were healthy, the load balancers were in good shape, and there was no noticeable change in network traffic. And since our business dashboards are only updated once per day, I wouldn’t have seen anything suspicious for almost 24 hours.

If User Support didn’t say anything to me, there would have been no way for me to know what was happening in real time. The metrics I was following closely, sometimes at the second granularity, were not good indicators of critical business changes that required immediate attention.

It was at this point I realized that something had to change.

Building an Alerts System to Improve Business Reporting

So, my team decided to build our own system to alert us when any critical business metric was in need of attention.

This is how we did it:

1. First, we listed indicators that were important for our business: account creation, new subscriptions (on each payment platform), successful automatic renewals, handling receipts of external providers (Google Play, Apple App Store), etc.
2. Then, we added monitoring usage data from anonymously collected data in order to get alerts per platform and release version.
3. Finally, we set up a process that alerted us after each production incident. That way, we’d know which incidents were covered and which were missing.

There was a problem though: The data for all those indicators was distributed across several sources.

For example, we use SQL databases for everything related to account creation, purchases, etc., but we use NoSQL databases for storing the anonymous usage logs.

A couple other obstacles we wanted to solve for were:

• We use an internal chat tool and ideally we would get alerts and notifications sent directly to us via chat.
• We couldn’t rely on yet another database structure for data collection and alerting, as that would mean additional maintenance and engineering resources that we don’t currently have (#startuplife).

So the decision was made to use ElastAlert, an open-source project by Yelp, that uses data in Elasticsearch for generating alerts. It also integrates well with different chat clients.

How Has It Worked Out?

We now use Elasticsearch extensively—we have a cluster handling more than a terabyte of data per week. Every engineer in the company uses Kibana to visualize the data collected by Logstash instances. For this project, we have a process that continuously polls our various databases and stores the alert data points every five minutes as Elasticsearch documents in a separate index. Elastalert then queries this data, uses the count aggregation feature of Elasticsearch, and is able to generate alerts. The stored documents have a reduced cardinality compared to the original data (hence some loss of information), and carry over pre-aggregated counts, but there’s enough information to make interesting alerts.

For instance, a typical document would be something like this: