How To Select Better Metrics: The DEM Principle
Header Image by @slotosch via Unsplash
Whether you are a product manager or a marketer, choosing the right metrics can feel overwhelming at times. There’s usually no shortage of things to measure, and you may often feel inclined to go with convenient metrics, such as off-the-shelf metrics available within your analytics tools.
A good example is website bounce rate or app MAU. I often see these metrics overused as KPIs simply because they're easy to find. That’s not to say they don’t have value. But sometimes they’re selected despite not being aligned to your current campaign or business objective.
When we choose metrics for convenience, this is a type of observation bias known as the drunkard's search principle, aka the streetlight effect. Simply put, the streetlight effect occurs when someone looks for answers in places that are most convenient (i.e. where the light shines brightest) rather than looking for answers in areas that may be most impactful, albeit more difficult to find.
Aside from convenience metrics, product managers and marketers can be susceptible to the allure of vanity metrics. If you're unfamiliar with the term, the team at Tableau defines this as "metrics that make you look good to others but do not help you understand your own performance in a way that informs future strategies."
As someone who has worked in the field of data for 10+ years, vanity metrics are something I've encountered often. It often comes down to fear of being transparent about mediocre performance or even outright failure. There are plenty of reasons why this dynamic can exist, but it's usually organizational culture, where managers or leadership do not create enough (or any) psychological safety.
A strong data culture grounded on psychological safety encourages people to experiment and try new things. But building a data-driven culture is no simple feat, and a topic for another article.
In this post, I wanted to cover a simple principle I use myself and with my clients that can help guide you toward better metric selection.
I call it the DEM principle.
Put simply, it’s a mental checklist where you ask yourself: are my metrics defendable, explainable and meaningful?
Let's unpack this further.
Selecting DEFENDABLE metrics
You have to be ready and able to defend why specific metrics were chosen as key success measures or KPIs. Time and again, I've watched marketing or product teams present a report to stakeholders, only to fumble when challenged by someone as to why certain metrics were selected over others. This can result in time wasted discussing or arguing about metrics, instead of focusing on insights and learning.
It's imperative that you select metrics directly aligned with your business objectives, and as such, you'll need to be prepared to justify why the metrics you/your team chose actually matter. Your stakeholders may not always understand the metric(s) at first; some may even disagree with you on whether a particular metric is relevant. But if you can't defend why you shortlisted specific metrics, you risk losing your stakeholders’ trust in the data you’re reporting.
Selecting EXPLAINABLE metrics
You also need to know how your chosen metrics work. Or, more specifically, where your metric came from and what it quantifies. I’ve been in so many meetings where a metric was being used as a KPI, but when asked what it measure specifically, no one had an answer.
This doesn’t mean you need to have a deep technical knowledge of how the metric is captured. But, you should have a detailed enough understanding to explain it to someone else in more than one way. I find this is a good standard because an expert usually has the ability to adapt an explanation of something complete to audiences of varying knowledge levels.
For example, you may find yourself having to explain a concept in a more detailed and technical way to one group, but then abstract it into a higher-level and more simplified explanation for another group who is less knowledgeable. I refer to this as traversing the levels of abstraction, a concept borrowed from academic Luciano Floridi. And it’s highly relevant to measurement, because you may need to explain how a metric works in different ways (e.g. more technical for engineers, more abstract for business users, etc).
Conversion rate (CR) is one of my favourite examples because this metric is often used, but many people don't know how to measure it. And that’s because there is no universal standard for CR, as it entirely depends on the channels you’re looking at and the user journey.
Let's say you have an online store and want to know the conversion rate of visitors to your site. If I asked you to define conversion rate in this context, you might say, "Well, it's the percentage of visitors who make a purchase." And you wouldn't be wrong. But when it comes to actually harnessing the data required to calculate the conversion rate, things get a bit more technical. For example, to calculate conversion, you would need to decide whether to use total transactions (i.e. including repeat customers) or unique transactions (i.e. unique customers who made a purchase) as the numerator for your calculation. Furthermore, you would need to decide what to base your denominator on, such as total website visits (not unique) or visitors (unique), and whether to base this on total site traffic versus traffic to a specific section of your site. Suppose you had paid media directing traffic to your website, such as a Facebook-promoted post. In that case, you could also consider using total reach or impressions from your ads as the denominator.
The point is that you need to know what a metric is counting and how it works to educate your stakeholders. And it doesn't matter if you're not a data analyst or data scientist. If you're in charge of choosing metrics to measure success, the buck stops with you.
Selecting MEANINGFUL metrics
Finally, you must ensure that you choose metrics that measure something of value. This starts with understanding your business objective(s) and choosing metrics that are directly aligned with this. Most importantly, selecting meaningful metrics requires you to seek truth and avoid the temptation of using convenience or vanity metrics.
Previously, I wrote about three types of personas that exist when it comes to using data to tell stories, which include the good (aka the truth seekers), the bad (aka the deceivers) and the ugly (aka the ignorant). A good analyst will always strive to be a truth seeker (i.e. the good), and every marketer or product manager should strive to be the same.
This is where convenience or vanity metrics often come into play. Sometimes, we’re highly incentivized to report on success, and only success. And I find that the expectation versus the reality of what progress looks like is often wrong.
An organization with a strong data culture knows how to encourage and motivate people to learn from mistakes and apply those learnings. A journey to success will always involve some failure along the way. The trick with conquering failure is to 1) recognize what caused it, and 2) apply those learnings to future launches or campaigns.
Choosing meaningful metrics can sometimes involve creating new metrics as well, something I call engineered metrics. Examples include creating custom metrics in Google Analytics, creating custom events in MixPanel, or even calculating a custom conversion rate in Excel.
No matter where your metric comes from, be it off-the-shelf or engineered, you need to ensure that it’s aligned with your business objectives and that it measures something of business value.
The DEM Principle in practice
The DEM Principles can serve as a helpful checklist that encourages you to question whether your chosen metrics or KPIs are fit-for-purpose. Asking questions like, "Can I defend why this metric was chosen?" Can I explain how this metric works?" and "Is this metric meaningfully aligned with my business objective?" These types of questions will put you on the path toward better metric selection.
However, a strong data culture and an organizational framework for choosing relevant metrics is needed to do this consistently across medium-to large sized teams.
To help with the latter, I've developed a campaign journey mapping methodology known as a Marketing Activation Plan or MAP for short. This approach helps you map out a campaign and identify the channels and assets you want your customers to engage with, as well as the desired action you want them to take. This can help you identify exactly what kind of content (i.e. assets) should be measured and how (i.e. actions).
My MAP methodology is currently available as an approved template in the Miroverse. If you're unfamiliar, Miro is a fantastic tool for visual planning and collaboration, and you can create a free account and duplicate my MAP board template using the link below.
If you enjoyed this post then you’ll love my marketing analytics course. Click the link below for more details.
