This is part two in the 2-part series on what it takes to implement high quality business performance monitoring. In part 1, The Building Blocks of Metrics Monitoring, we discussed key concepts that are prerequisites to success. In this post we will cover the stages a company goes through along its journey to achieve metric monitoring nirvana.
The Ascending Stages of Difficulty in Metric Monitoring
Stage |
Infrastructure Quality |
Data Observability |
Root Cause Analysis |
| 0: Data blind | LOW | N/A | N/A |
| 1: Data initiated | MED | LOW | N/A |
| 2: Quality reporting achieved | HIGH | MED | LOW |
| 3: Alerting & RCA initiated | HIGH | HIGH | MED |
| 4: Alerting & RCA achieved | HIGH | HIGH | HIGH |
Table 1: To achieve success in metric monitoring, organizations must steadily improve their infrastructure quality, data observability, and RCA abilities.
For definitions please refer to part 1 of this series.
Stage 0: Data blind
When we consider the landscape of all businesses, which includes the longtail of small business (online and offline), the majority are likely not putting their “data” to use. They typically only look at the high-level metrics such as revenue, costs and profits. They are unlikely to be collecting data from their distributors, building reports out of their line-item accounting P/L statements, or visualizing the ROI on any marketing spend.
Why companies get stuck here: Gut based decision making is often considered good enough. Analysis is tedious and leaders don’t see the benefit.
Stage 1: Data initiated
Most businesses, be it an Etsy shop or a detergent supplier to Walmart, want to leverage their data to grow sales, streamline expenses, or generally speaking – better manage their business. Data-initiated companies are the ones that have leveraged their data assets (could be google analytics or even an annual sales pdf) and built at least one ad-hoc report. This initiation is like pulling off a blind-fold to see a hidden world, one that so far you could feel and hear but not see. Companies rapidly advance from this stage because once they’ve had a taste of applied analytics they are eager to make it a part of their process.
Why companies get stuck here: Business feel satisfied with doing the bare minimum. Hiring analysts can be expensive.
Stage 2: Quality reporting achieved
Establishing an intermediate level of analytics monitoring sophistication – where you have pretty dashboards that highlight key metrics and are viewed occasionally – is achievable with moderate effort. A reliable data source, some data transformations, and a visualization tool will get you most of the way.
Why companies get stuck here: Unfortunately, this is where most companies plateau in their maturity. Maintaining reliable data is notoriously difficult, building reports is time-intensive, and existing analytics solutions haven’t innovated beyond their core value proposition.
Stage 3: Alerting and RCA initiated
Teams who reach quality reporting at stage 2 become a victim of their own success, they now have a new problem – fire drills. A dashboard gets updated, a few days later a manager reviews it to find that metric has fallen… and alarms go off.
Many teams will try to address this by using the built-in alerting mechanisms (like GA alerts or Tableau digests). This helps, but falls far short of internal aspirations. This is where I got stuck during my time at Nest. Either the alerts are irrelevant and too frequent or the pre-scheduled email digests are too boring to pay attention to. At this point, industrious and committed teams may explore alternative analytics service providers or attempt to engineer an in-house solution. YMMV.
Why companies get stuck here: Creating high signal-to-noise alerts that add value requires advanced ML and engineering chops. Business and Product teams are not equipped to build new internal tooling that does this.
Stage 4: Alerting and RCA achieved
Analytics auto-pilot. This is a dream many aspire to but few achieve – the equivalent of having PagerDuty for business metrics. It’s like buying insurance, you know that in case of an emergency you’re covered. You will receive news of trends and incidents in performance as they happen, when you receive an alert, you’ll know it’s important. Your system will have analyzed all the possible business drivers – a combinatorially large dimension space – and it will identify the most promising causes. You will sleep soundly, confident that your metrics are being monitored and that you know for sure what is and isn’t driving performance.
Fig 1: The road to business performance and metric monitoring nirvana. Can you identify where your organization is on this timeline?
Conclusion
Analytics today is designed to confirm the expected (to show business as usual). Not to spot the unexpected. Detecting the unusual requires keen business savvy or inordinate effort because the size of data is exploding and you are looking for the proverbial needle in a haystack.
Few companies have the time, resources, and priorities to build out bespoke analytics tooling. But as data becomes abundant, processing becomes cheaper, algorithms improve, and Analytics as an industry matures, we can turn to our elder cousin, Application Performance Management, to glimpse where Analytics is going. If you are interested in leapfrogging your business to stage 4, reach out to us at sales@outoftheblue.ai.
