Reading Time: 6 Minutes
As digital analysts (or people who manage digital analysts) we have a pretty cushy job. We get to play with data all day, make pretty charts and visualizations and tell other teams how well or poor they're doing. Ok, so that's not entirely true, but the reality is most analytics teams spend most of their time helping others justify their performance and impact without considering the ROI of the analytics function itself. Avinash Kaushik wrote about this a while back where he proposed a general approach to measuring the return on analytics (ROA). This is a great topic and one that I think will become more prevalent in the coming years. Today, most analytics teams likely don't have a great deal of pressure to justify the investment in their role. But as analytics becomes less of a shiny new toy and more of a standard must-have function, C-suites will undoubtedly turn to analytics leads to start justifying what business value and impact they're generating for the organization. Kaushik's article on ROA is a great starting place to think about business value, but I want to talk more about diagnostic measures, or optimization, when it comes to the value analytics creates for the company.
I recently came across this post by Cheryl Weibe who leads the Manufacturing Applied Analytics team for Teradata’s consulting business. In her post she included a few charts from Rexer Analytics' 2013 Data Miner Survey. This really sparked my interest as finding new and innovative ways to reduce the time it takes you to analyse your data, and subsequently, to action on your data, has been on my mind lately.
From experience I've found that we tend to over complicate how we communicate or interpret data when it comes to reporting. Time and time again I see 60+ slide PowerPoint decks and Excel documents with endless numbers of sheets that are heavy on data, but end up saying very little at all.
There are lots of ways you can think about optimising your measurement program, such as auditing your current reporting landscape by asking questions like:
- How many reports are you sending out?
- Who do these reports go?
- How frequently do the stakeholders need or want the data, and in what format or medium?
- What are the sources of the data?
Beyond auditing what's currently taking place (the AS-IS), you can also benefit by reviewing the reporting stakeholder needs by asking questions like:
- How will they use the data?
- How frequently will they take action?
- How do they want to receive the data?
- How do they want to consume/interpret the data (e.g. more/less data, more visualization vs more raw data in tables, etc)?
Reviewing/auditing a measurement program can be quite a large effort and I'll cover this in more detail in another post. For now I just wanted to leave you with some thought starters around how we can assess the impact that analytics is having on your business.
Figure 1 below shows the results of a survey conducted by Rexer Analytics where they found that a majority of companies (33%) reported being able to analyse their data [presumably] within days of it being generated. If you find yourself located more to the right side of this chart where it takes you (or your agency) weeks or more to analyse your data and generate insight you may need to reconsider your approach to measurement and reporting.
I think this is a great study, but I'm going to propose a slight change to the wording so our KPI is called Time to Insight rather than Time to Analysis. I have a very specific definition for insight, that is, an interpretation of data which has value and enables a business decision. That's an important distinction because I don't think analysis always deliver insight.
Figure 1. Time to Data Analysis
After analysing your data you should consider measuring the average times it takes you to take action. This is a tricky one, as the type of action can have a huge impact on the time it takes to deploy (e.g. optimising ad copy vs your website's UX/design). So you would do well to start by first identifying the types of optimization (or actions) you would commonly take as a result of analyzing your data (e.g. optimizing copy, creative assets, UX, spend, targeting, etc).
If you look at figure 2 below you can see that the majority of companies surveyed reported being able to take action within weeks. For the purpose of measuring the impact of analytics, I would even propose separating action from deployment, as taking the steps to act does not always result in a successful deployment. So now we've got Time to Action and Time to Deployment.
Figure 2. Time to Action
Overall we have three great potential diagnostic KPIs to help you measure the impact your analytics practice is generating for the company:
- Time to Insight - The average time it takes you to produce an actionable insight from the point at which your data is generated.
- Time to Action - The average time it takes you to take action from the point at which your data is generated (or you could base this on the time at which the insight was generated). Also, action for me would be defined as something like deciding on the correct course of action and agreeing to commit to making the change or optimization.
- Time to Deployment - The average time it takes you to successfully deploy a change from the point at which your data is generated (or you could base on the time at which the insight was generated or when the action was initiated).
Your end goal with measurement should always be based around accomplishing more with less. Simply put, you want to move your business toward the left of both charts above. The faster you can collect, analyse and produce insights through lean reporting the less time you end up producing bloated reports that that end up in the trash. And better yet, the faster you deliver those insights, the faster you can benefit from them.
As mentioned above, I believe it's just a matter of time before analytics managers and leaders are put under the same microscope as other business units to justify their existence. As an analyst, manager or business lead you would certainly benefit by staying ahead of the curb and investing in these types of internal measurement practices now. Time to Insight, Action and Deployment, as well as ROA are just a few ways to quantify the impact of analytics. But the best metrics and overall approach for your company will depend on you, your reporting stakeholder's needs and your organizational culture.
I know investing in internal analytics impact measures might be a tough pill to swallow. It will almost certainly put you and your team under a new lens, create added pressure, will require buy-in from the executive branch, and will undoubtedly require some investment to get it up and running. So why measure the impact of analytics? Simple, investing in this type of measurement will help future-proof your organizations investment in analytics by ensuring the right measures are in place for tracking performance and knowing when analytics is generating less impact. Also, improving metrics like time to insight, action and deployment can also help tell a great story to management, that is, you're achieving more (impact) with less (time, money, resources).