analytics business-strategy data-science entrepreneurship management product-management

Getting the Product Usage Data You Need — Part 3 of Transforming Your Product Team’s Analytics Prowess

If you’re joining me for the first time, this is a series on evolving the analytical abilities of your Product team. The first post covered data culture, while second focused on the fundamental data that you need as a Product team. This post will double-click into where to get that fundamental data. After reading this, you will be able to:

  • Locate where different kinds of data are commonly stored. Usage data will likely be in a different place than client information or qualitative product feedback.
  • Identify who likely has access to data you need. You or your engineers might not own the data. That’s okay.
  • Shorten the time it takes to get such data. Compiling data can be one of the most time-consuming part of analytics.

Goal: Getting to Insightful Answers Quickly

The mission is to provide insightful answers to fundamental questions within minutes. To do so, you will need fast access to — or ownership of — the relevant data sources.

Four weeks into my Growth Hackership at Cornerstone, our CEO stops by my desk and asks me, “What’s the adoption for the new Admin Console?”. It was Monday. Admin Console was a new feature released on the previous Friday (just 60 hours prior). I put my head down and started number crunching…

Sidebar: During my time on the Xbox Strategy team, it was common for executives to ask similar (arguably, harder) questions on any given day. We would pull two to three resources (sometimes referred to as people) to build a 15–20 slide analysis. The deck would include several models (financials, economic projections, market share, etc.), a stereotypical appendix, preliminary recommendations, and a big shiny “Draft” annotation. This was all delivered within hours of the original question. Resources (aka: people) would pull data from our BI, Finance, Marketing and marketing teams. We would also go through market research, equity research, and our networks. It was beautiful chaos and the analyses were superb.

No analyst, manager, director or GM worth their salt would simply blurt out a number to the Executive. That was a definitive way to get resourced on to shit projects. Eventually, the directors would take you out back and move you to a completely different team. The work had to be good. And fast.

7 hours later, I sent my paltry follow-up email to our CEO. (Note: all correspondences are paraphrased) “It looks like we have 10% of our clients that have ‘adopted’ the new feature. I’m not sure how they’re using it, but they at least loaded the page.” So insightful, right? Wrong. You’d think I was lazy with an answer like that, but that would be far from the truth. I’d spent the last 7 hours running into dead ends.

  • The PM didn’t have any data. She in turn asked her devs.
  • Devs didn’t have time to write the query. The devs said a DBA needed to run the query, but they’d need to write the query first.
  • Only DBAs could run queries on Production databases. Devs needed to write the query first. Or I could try writing it myself.
  • The DBA said it would take five days to get results back. No point writing it myself. I needed an answer quickly.
  • Dove into our machine logs (Splunk). I looked through web traffic logs where I was able to count unique clients loading the page. I finally had something. 10%.

The response came in. “10%? That’s it? It should be 100%.” Spoken like a true executive. In any case, this entire exchange was underwhelming. It took me 7 hours to find and share one data point (10% “adoption”) with practically no insight.

Man, this was going to be a long analytics journey at Cornerstone.

The moral of the story is this: insightful data also needs to be timely. Decisions will be made with or without the data you’re looking for.

1. Where will this fundamental data come from?

Let’s recap the framework that I introduced in the previous post. There are three core questions that Product teams should be able to answer:

  • Are your customers using it meaningfully? (Adoption)
  • Is their usage “on-script and on-schedule”? (Engagement and Retention)
  • Are they satisfied with your product? (Satisfaction)

For simplicity, let’s refer to adoption, engagement and retention data as Usage Data. Once you have defined desired user behavior, you can start gathering data. Let’s dig into where to find the data and who might be able to give you access.

1.1 Usage Data

Usage data is in the domain of Product and Engineering. With the exception of Accounts information, Usage data will sit in databases, logs, or tracking systems that Product and Engineering manage. It should be the most readily accessible source for PMs.

Sufficient usage data comprises of three categories:

  • Events — These are things that happen or things that users do. Events are important to track because they describe activity. Activity, in turn, tells us about value creation.
  • Users — These are the subjects of events. They are the ones doing an activity or are subject to a system event. We will want to describe users in relevant detail. We will want to be able to understand how product usage changes across different groups of users.
  • Accounts — These are your clients (companies or organizations) that buy your products. Depending on your business, this may be one and the same as Users. If you distinguish between the two, you can analyze company behaviors by industry, size, or other dimensions.

Events and Users data are stored and accessed through databases (e.g., SQL), logs (IIS, service logs, database calls, etc.), or event tracking platforms. While databases or logs may store have of what you need, the data will be in a raw format. You might need to brush up on SQL to extract the data and practice Excel or Tableau to get your insights. Event tracking platforms will make data consumption easier with out-of-box reporting and charts. Examples of event tracking platforms include Amplitude (❤), Pendo, Google Analytics, and MixPanel.

Accounts information will normally be in your CRM, like Salesforce. You can reach out to business analysts from Sales Ops for access to this. You might also consider partnering with folks from Marketing or Biz Ops. In the ideal case, you’ll have read access to pull information any time you need. If governance is tight, start with requesting snapshots and exports.

1.2 Satisfaction Data

Satisfaction data is normally owned by the Customer Experience (CX) teams. This experience data measures the quality of the relationship you have with your clients. While Product may not manage these systems, it’s critical to have unobstructed access to your CX data.

Ask your CX team to create a dashboard for the Product team. It should include snapshots as well as historical trends. Text analytics is useful to understand the themes and concepts that clients are taking about. Ask for those, if available.

You’ll want to get a well-rounded view of the client experience. You need qualitative feedback from surveys and quantitative measures like NPS, CSAT and CES. Systems like Qualtrics and Zendesk have thorough CX management capabilities. Like product analytics platforms, they make data capture and analytics easier to do.

A few principles to keep in mind when capturing and reviewing Satisfaction data:

  • Sample responsibly — I shared a story in my last post about how our Customer Success team cherry-picked respondents for surveys. Doing that raises integrity concerns beyond the data. Moreover, you deny the organization something very valuable: the truth.
  • Remove bias from the questions — As simple as this sounds, it’s very difficult to do in practice. We’re asking people to share their thoughts and feelings. Phrasing can sway responses. Questions like “ Do you think the design of _____ is intuitive?” No bueno.
  • Get enough volume — The number of respondents changes insights from directional to significant. Response rates for satisfaction surveys are about 10%-30%. Getting a low response rate may mean low engagement from your customers, and can be a data point in itself.
  • Be honest — If you have a high NPS score (around +40 for B2B SaaS)… Cool, keep it up. But, if your CSAT or NPS scores kind of suck, face it and roll your sleeves up. Problems arise when teams downplay or ignore negative data. That’s a wonderful source of growth.
  • Keep it fresh — Make sure you have a steady stream of CX data coming in. CSAT is transactional and can help you track operations. NPS is relational and a lagging indicator of how a customer holistically feels about you. Either way, make sure you’re able to watch trends over time.

2. Get Your Data Faster

As a reminder, our goal here is to cut the time it takes to get raw data. Compiling and prepping data is often the most time-consuming aspect of any analysis. Below is a list of common scenarios and ways through which you can secure data faster.

2.1 You have the usage data in back-end systems

I’m sure that you have a rich variety of usage data sitting in databases and logs already.

  • Request read access and pull it yourself (duh). This means you’ll need to know SQL or the relevant query language. This also means you need to know the data structure and database schema well. This is a highly flexible way, but complex and technical.
  • Design scheduled reports with your devs. This is healthy compromise between automation and minimal load on engineering. Sit down with your engineers and map out the output that you need. Schedule that query to run on a regular cadence. The downsides are (a) ad-hoc queries are not supported; (b) exploratory analyses are difficult with the fixed report; and © report changes require dev effort.
  • Blend and visualize data in an analytics platform. Platforms like Alteryx, Power BI, and Tableau are powerful and easy to use to blend data. You can marry live SQL data with other sources, conduct your analysis and visualize. This is the most dynamic way to access your data with minimal dev effort. It just costs money.

2.2 You don’t have any data

Oh, dear. Keep reading. The next section should help.

2.3 You want to get better usage data moving forward

There’s really only one way to do this: Instrument.

  • Instrument for usage data with an event tracking platform. We’ll go into the details of instrumentation in a later post. But, the idea here is to (a) know the specific events you want to track; (b) invest in an event tracking platform; and (c ) work with engineering to instrument. The pay off here is in having purposeful data that you can access any time. Depending on the platform, you may need to do the analysis or visualization in another tool. But you got the time-consuming part out of the way.
  • Instrument for CX and satisfaction data with in-product surveys. Many providers like Qualtrics and Pendo allow you to embed surveys into the product. You can set those as passive surveys that ask for CSAT or NPS. Or, you can trigger the questions to appear after a specific user action. Make sure you have access to view the results if the data still goes to the CX team.

3. Don’t be cheap. Invest.

Getting quality data and getting it fast isn’t free. It requires investment mostly in the form of time. It takes time to figure out what you want. It takes time to build the relationships to get access. It takes time to write queries or instrument the product. This is a given.

There’s the other form of investment: money. Monetary investment accelerates speed-to-data by orders of magnitude. We used to spend days to get to a handful of adoption insights. With a product analytics platform, we have been able to improve prioritization decisions. We also save thousands of hours of development on unimportant and non-urgent work.

Before you get budget for a “huge” buy, you’ll need to justify the cost. Most vendors provide free trials of their platform. That should help you get a sense of the implementation cost. At the same time, try to get data the (usual) hard way. Compare the two methods and pitch the difference to your stakeholders. The value of an analytics platform becomes clear through speed, quality, depth, and insight.

Sidebar: After that experience with the CEO, I put together a proposal to evaluate a couple product analytics vendors. You can view my proposal here: Feel free to use this as a template or inspiration for your own pitch.

No promises on a fast decision or commitment. It took me two years to convince our organization to invest in Amplitude. There was major hesitation at high levels of leadership in spending money on a solution. I spent 3 hours a week for ten months with Amplitude to convince Cornerstone of the need. We compiled case studies, organized training sessions, hosted demos, and built ROI models.

I got the most support from our VP of Business Operations (not my boss). He was gracious enough to try clearing budget for me. But that wasn’t enough to get it through. The breakthrough came when we acquired a company and brought in new leadership. The incoming executive became my new boss as VP of Product Strategy. He was able to push it through after several rational conversations with our CEO. (Thanks, Reed!)

The Payoff

Product teams will struggle with an analytical renaissance if data is hard to come by. Investment — time and money — facilitates that evolution.

After years of campaigning, we finally got Amplitude Analytics. It’s been worth it. We’ve paid off our annual license fee with some of the prioritization decisions alone. We consistently justify product designs and deprecations based on user behavior. We are able to do ad-hoc analytics in minutes. We are able provide a full-story to executives on product successes. We are able to assess product adoption in real-time and spin up client evangelism programs within days. The investment made has accelerated the evolution of our Product team.

Some keys takeaways:

  • You probably already have some usage data. Try automating reports.
  • Instrument to get the full picture. That is Events, Users, and Accounts.
  • Make friends with your CX team to get satisfaction data.
  • Satisfaction data is volatile. Capture it responsibly and with integrity.
  • Invest time to make data accessible within minutes.
  • Less time pulling data = More time generating insight.
  • Make a business case to increase investment. Use this template.
  • Be patient. It may take a while. Trust me.