The Analytics Requirements Document

Launch and pray doesn't work when it comes to data.

While I mentioned the Analytics Requirements Document (ARD) as a concept in a previous post, I really spoke about it in depth during a talk several weeks ago (slides, recording). The content from the talk has been transformed into this post and elaborated upon.

---

The Head of Product at your e-commerce company comes to you, an analytics lead, and asks: “so when are we going to get performance metrics on that rewards program we launched last week?”

You pretend you know what’s happening (there was a launch?) and say by end of the week you’ll have something (requirements might be complex, TBD). You fumbled through that one, but now the real work begins. Upon approaching your manager, the Head of Analytics, you figure out they knew about the new rewards program but didn’t realize it’s already out in the wild. The company does need to make a decision on whether to extend the rewards program to a broader set of users, though, so time is tight.

Some cats get herded and you get some set of requirements around how to make that decision, requirements that may or may not make sense. Another week has gone by, but you can now start doing your job: delivering data around product launch metrics.

Within 30 minutes of doing the actual work, alarm bells go off in your head. There’s a table listing rewards used on a particular day by a particular user, but rewards don’t tie back to items. You can’t actually know if users are building and using rewards points in a way that’a beneficial to the business (like making sure they’re not getting rewards for returned items).

This probably sounds familiar, so let me be succinct with what’s to follow: this news gets back to the Head of Product and the CEO, both furious that two weeks are wasted on a product that can’t be measured. The data and product teams coordinate to actually generate useful data (you just need a foreign key on the item if it was purchased using rewards points). Everyone goes back to what they were doing, until this happens again next quarter with the next big product launch.

It doesn’t have to be this way. Analytics should be a first-class citizen when crafting product launches.

A virtual ribbon cutting: let’s talk contracts

Source

How do we coordinate across parties, teams, and interests when it comes to launching anything? Consider real estate. We all know this red ribbon when a real estate mogul opens the doors to a new building, not dissimilar from an imaginary ribbon cutting when an engineer deploys a new feature to production.

Leading up to any ribbon cutting are a set of agreements.

In real estate, a launch of a new development doesn’t happen without preparing some percent of the units to be leased. Whether it’s consumers or businesses, leases are signed and legally binding contracts. This is the most formal version of a contract, in any sense of the word.

Similarly for product launches within a company, product teams have agreements with the rest of the business of what the launch entails. Usually in the form of a Product Requirements Document (PRD), a product manager outlines exactly:

• What the engineers are to build

• Why this is important for the business

• How other teams will be affected by the launch

There’s been a lot of talk around technical data contracts in the community when it comes to ensuring data output meets certain criteria, like schema format (for example: buz.dev, check it out). These types of data contracts are between engineering teams and consumers of data, usually analytics teams. While an important step in ensuring engineering and analytics teams are aligned, coded data contracts exclude other parts of the business like product, marketing, and customer success teams who function as owners or supporters of the launch.

Everyone has expectations from analytics, and analytics has expectations of everyone else. If product has process, why shouldn’t analytics?

The ARD: what it is and why you need it

Going back to our rewards program example, the only reason for launching a rewards program is to benefit the business. In theory, these types of programs create more sticky users that end up generating more revenue for the company while they may or may not redeem their points in time. While I generate an borderline unhealthy amount of points on Backcountry.com right before ski season, I forget about these points half of the time before they expire.

All of this impacts the performance delta of the lifetime value of users that opt into the rewards program compared to ones that don’t. Ah, lifetime value: a metric. It has to be segmented on revenue generated from points versus actual dollars, and on users depending on different types of subscribers to the program.

Before launching the rewards program, it would be prudent to outline how data related to the performance metric is actually going to be stored and analyzed instead of following a “launch and pray” model.

The ARD should be generated by the analytics team in parallel to a PRD early in the product evolution lifecycle, outlining tracking expectations.

The ARD and PRD are quite similar in purpose and design. Let’s walk through the ARD section by section. You can follow along with my example ARD here.

1️⃣ Goals. Just like a PRD, this should confirm both the goals of the product and what metrics are important for success. This section will guide the content for the rest of the document. Most importantly, outline which metrics are needed to track product goals from the PRD.

2️⃣ Data criteria. This isn’t criteria for the product itself—rather, what data needs to exist to determine whether or not goals are met. Be as specific as possible here; instead of tracking in aggregate, define what specific relationships are needed in the data. Explain what you need, not how to implement it (that’s material for an RFC, see below).

3️⃣ Assumptions. In the e-commerce example, I mentioned tracking rewards used for specific items, which assumes item level order data exists. While this isn’t a lofty assumption, consider other assumptions that may exist surrounding financial data, marketing attribution, and product analytics. The purpose of this section is to outline all assumptions and ensure they’re not known to be false.

📝 Side note: participating in the RFC process. While this shouldn’t be part of the ARD as it’s a process usually spun up by the engineering team, it is extremely important the analytics team by involved. In an RFC (Request For Comments), one engineer will put together a proposed architecture while other engineers are invited to make suggestions. The goal of analytics being involved in these meetings is to make sure it’s not only possible but also not excruciatingly difficult to track metrics. For instance, any data engineer might take issue with a foreign key being stored in a JSON object instead of its own field.

4️⃣ Out of scope. Equally as important as the goals section, this should set expectations on what isn’t tracked. If the analytics team outlines what isn’t possible from the get-go, it avoids miscommunication or wasted time if missed requirements are caught early.

5️⃣ Stakeholder sign-off. This is the most important piece. ARDs shouldn’t be proposed in a vacuum. While consensus can be achieved verbally, putting a signature on a document usually triggers everyone to read it in more detail. Sign-off from relevant parties leads to alignment and ownership across product, engineering, and business teams (usually marketing and customer success are involved).

Final thoughts

Source

Just like product has a seat at the table for most engineering prioritization meetings, analytics teams need a seat at the table while the company outlines new features. Every launch should be measured, likely using both first- and third-party data. Data should be used to gain insight to assist in decision making, and the only way to do so is involving analytics teams early (check out Emilie’s Coalesce talk for more on this).

A word you may hate but can’t live without: process. As analytics teams evolve, so must their process and formalities. Product is a great example of a team that needs alignment to meet goals and be involved in the process of the product lifecycle (check out Abhi’s Coalesce talk on modern data teams and analytics requirements).

To successfully reach alignment across both technical and business teams, we need coded contracts and process in tandem.

---

Thanks for reading! I talk all things product, marketing, data and startups so please reach out. I’d love to hear from you.