Let’s assume that you are aware of product analytics and its many benefits. You’ve figured out what questions are important for your Product team to answer. You must answer these questions. How would know that you’re really creating value for your customers? You need concrete proof and you need it fast. This is where a product analytics platform comes into play.

I recently went through a two-year journey at Cornerstone OnDemand to get a proper analytics platform. It was a Zero-to-One transformation. I figured some of you might benefit from my reflection.

This article will cover the five steps to procuring a product analytics platform. (This framework should work for any other systems you want to purchase.)

The steps are:

1. Define exactly what you need
2. Commission a trial
3. Evangelize internally with savvy
4. Start the formal procurement
5. Envision your operational plan

Ask yourself “Do we really need a platform?”

In previous posts, I covered (1) the fundamental data PMs need, (2) the need to get that data quickly.

To quantify value creation, PMs need to provide proof of well-defined usage and satisfaction. That insight should be readily available within minutes of inquiry. If your organization cannot deliver answers with speed and reliability, you should invest in a product analytics platform.

Three years ago, it would take several hours for the savviest PM to find the MAUs (Monthly Active Users) for a particular product. This morning, I watched a lead PM arrive at the same insight within 15 seconds (5 clicks). That’s two orders of magnitude improvement in speed.

If any of the following are true, you should invest in a product analytics platform:

• It takes hours, days, or weeks to answer basic questions about user counts
• You can answer if, but not how, a product is being used
• You distinguish different types of users of your product
• Your data is unreliable because of inconsistencies or incompleteness
• You have blind spots of how a user moves through your product
• You rely heavily on other teams, including engineering, to get data
• You don’t have any data at all

To be clear, data from analytics platforms aren’t a silver bullet. But it is a catalyst for better decisions and business enabler.

Step 1: Define Exactly What You Need

The universe of available product analytics platforms is vast. You’ll need to be clear about what your organization requires. Some teams may build their own home-brew tracker. Others will pay for enterprise-grade platforms complete with machine learning and external data platform integrations. Then, there’s everything in between. Here’s a framework you can use to define exactly what you need.

Evaluation Criteria

• Core Capabilities — Do you need a platform for visualization-only, data capture-only, or data capture + visualization? If you have reliable, fast access to product data, then you might just swing for an analytics platform like Tableau or Alteryx. If you need data, you’ll opt for one of the other two options.
• Scale — How many events do you expect to track every month? Most important is to estimate the order of magnitude — hundreds, thousands, tens of millions, etc.
• Auto-track vs. Precision tracking — Several vendors make data capture as simple as placing a JavaScript snippet in your footer. These will track any and all activity in your product. There is a trade-off for this completeness and ease of install. Auto-track generates a lot of data, which can be overwhelming and possibly noisy. Also, it’s harder to capture more descriptive information about each event. That’s where precision-tracking shines. The PM can define specifically which events to track and also which metadata to pass along with it. The trade-off here is that developers will need to extra spend time instrumenting.
• Data Export options — Will you need to export the event data in bulk? Do you want a vendor that will ETL your data into another platform? These might be more advanced considerations, but they’re worth exploring if you need some flexibility in the future.
• Integrations — An analytics platform is even more potent if you can integrate it with other business systems that you’re organization is using. At Cornerstone, we use the Salesforce integration to pull in Account object information. That makes it easier for us to analyze our product data using Salesforce attributes such as industry or client size. In other instances, you can integrate product usage with A/B testing platforms like Optimizely.

Sidebar: With Cornerstone, my main challenge was to raise the analytical maturity of a 30+ Product team. We were all over the place in terms of analytical literacy and also far behind on data availability. I searched for a platform that met the following criteria: (1) Data Capture + Visualization; (2) Supported hundreds of millions of events monthly; (3) Precision tracking so we don’t introduce noise and confusion to the team; and (4) Integrated with Salesforce. For a team that size, change management is a pain. Governance was the most important factor that I looked for in a platform.

Step 2: Commission a Trial

In this step, you want to do a bake-off between potential platforms. The requirements that you put together will help you shortlist potential vendors.

Step 2.1: Secure demos with your vendor shortlist

Schedule 30-minute demos with the vendors you’re most interested in. Pay attention to these things during the demos.

• The vendor’s philosophy — do they focus on user behaviour analytics or simple activity tracking? Data Quantity vs. Data Quality?
• Use case coverage — is the platform designed to answer the questions you’re asking?
• Instrumentation effort — how much development effort is required to generate data.
• UX and UI ease of use — you really should find a platform easy to use for conducting analyses.
• Training and support available — this is extremely important for a successful implementation.

Once you identify the platforms that you like, request a trial. They usually offer 30-day trials. For most organizations, that should be enough time to sense value.

Step 2.2: Build an Exploratory Team to spearhead the trial

A friend / Technical PM recently asked me, “[who] should spearhead getting a more robust analytics platform/plan?” Most organizations think that evaluating platforms will take significant time and overhead. In reality, you just need a small exploratory team of three people:

• Product Manager Lead / Director — Their primary role is to evangelize the benefits of a platform. They also need to influence the right stakeholders for investment.
• Product Manager (IC) — The PM is the linchpin to this trial. As the primary user of the platform, she can evaluate the usefulness of any systems in the trial. Following that, she will also be the best case study when a formal pitch is being made internally.
• Developer — The developer will measure how much effort it takes to instrument the product. He can also offer technical insights such as latency impact and security concerns. The dev will be a great partner in documenting the instrumentation process for other engineers.

In some cases, the influencing PM and the hands-on PM can be the same person.

Step 2.3: Make the pitch and secure an executive sponsor

In large or bureaucratic organizations, purchasing decisions can take a while. You’ll want to pitch and create awareness of your product analytics initiative as soon as you identify the need.

In your proposal, you’ll want to cover the following:

• The Need — What data is missing and what that’s impacting,
• Context — What do you have today and their limitations
• Proposed Solution — What you’re looking for and which vendors provide
• Proposed Trial Timeline — Outline the process, target dates, and the exploratory team
• Evaluation Criteria — On what basis will you evaluate different platforms
• Next Steps — What you need from them now and what will happen after the trial

The sooner you can secure an executive sponsor (e.g., VP Product), the faster you can start evangelizing . Just make sure your sponsor can actually back you up and understands what you’re trying to do.

Step 2.4: Instrument and Experiment

As the PM, the most important part for you is to design your experiment well. Think of an analysis that is currently difficult or frustrating to do. It should represent an analysis that is both common to PMs and impactful for decisions. Instrument to solve this frustration.

Think of the questions you need to answer. Define the measurements that you’ll use to evaluate, so you can compare the old way vs. the platform.

At this point, you should make a taxonomy of events and the attributes that describe them. The taxonomy standardizes the definition of each event and specifies when to trigger it. The vendor should provide documentation or a person to help you form this.

With taxonomy in hand, the developer should be ready to instrument the product. He should pay attention to the effort and complexity of instrumenting. The data should start flowing as users interact with the instrumented features.

You can start your analysis once you have sufficient data. There are two goals here. First, you want understand what it takes to replicate results from the old analysis. This allows you to compare effectiveness of the platform against the old way.

The second goal is to generate more insights with the data you’ve captured. You can showcase the additional value that’s created by the platform through exploratory or explanatory analyses. It shows that you — with the platform — can go above and beyond to make the best product decisions possible.

Step 2.5: Synthesize your results

The point is to offer objective results of your trial with the vendors, so you can proceed to the next steps. You’ll want to highlight the learnings, the benefits, the overhead, and other costs. At its core, you want to share whether your experiment was successful or not.

Step 3: Evangelize Internally and build your case

The goal of this step is to grease the wheel for a procurement. You can achieve this by providing business justification and an initial operational plan. You can start while trying out different platforms. You need stakeholders to buy into the value of a product analytics platform — not the offering of specific vendor.

Build your coalition of stakeholders

You can build your coalition in waves, with different messages for each.

The first wave is the primary beneficiaries and implementation teams. This stakeholder set includes Product leadership and Engineering leadership. You’ll want to bring them on board as soon as possible. The message should focus on faster decision-making towards delivering better products.

The second wave of stakeholders should be other beneficiaries like client success and marketing. The message here should focus on new insights that can advance their programs. You can share user behaviour and success statistics to improve client engagement or bolster sales messaging.

The last wave of stakeholders will be procurement and finance. The message for this group is about the value you bring to the broader organization. You’ll want to cover the horizon over which you’ll recoup your investment.

Leverage vendor sales assets to evangelize internally

There are several assets from vendors that you’ll want to share. These assets help you build your case for investment.

Ask for these things from each vendor you’re interested in:

• Scheduled or recorded demos
• Pitch decks that cover their philosophy, capabilities, and support
• Case studies especially of clients that have a similar context to yours
• Reference customers that you can call
• Licensing and pricing details
• Roll-out or implementation support plan
• Ongoing support packages

You might need to drip these or repackage these assets to make them more consumable. In the meantime, make sure you’re also comfortable with the content.

Be patient. It might take a few meetings to convince them.

There are so many reasons that your pursuit could fall through. There may not be budget for it. The data culture may not exist. Some people may just not understand it.

If you’ve done the work in building a coalition, shared business justification, and run a successful trial, you have everything you need. Everything else will be objections that you’ll have to address as they come up.

If everything checks out, it’s time to buy.

Step 4: Start Formal Procurement

Figure out costs and budget

As you made your way through step 3, you should have a sense of how much budget is required. Most vendors will break down the cost by a platform fee, the number of licenses (seats), and/or data volume.

You will inevitably be asked, “How much will this cost?”. You should lead with the cost levers and provide the cost of a few scenarios. Namely,

• Minimum viable insight — something that allows you to measure the success of a high-impact feature or product
• An entire product or feature set — This allows you to understand user behaviour within a specific product or set of use cases
• All products — This is the nirvana state of full instrumentation. Use this to bookend and show how the investment scales

The budget may come from IT, Product, or from your coalition chipping in. Start with your leadership and branch out as necessary.

Engaging with procurement

Depending on the size of your organization, you may have a formal procurement team. If you have a procurement team, you can share with them the context and justification. You should also have a sense of budget and the requirements.

Procurement’s main objective will be to negotiate with the vendors to achieve the best possible deal, and to work towards a signed agreement.

Take a backseat and be ready to answer any questions as they come up.

Step 5: Envision your Operational Plan

Assuming the negotiation goes well and that you’re close to securing a shiny new analytics platform, it’s time to build out your roll-out plan.

Be aware of these contract terms:

• Maximum event volume in the contract
• Number of seats/platform licenses
• The term length of the agreement
• Support level

The contract terms give you a set of constraints to operate under.

Roll-out Plan Components:

• The first product(s) to instrument — I strongly suggest that you start with a small but dedicated team. This minimizes execution overhead. The 80/20 principle applies here. You will learn best practices with a small team, which you can distribute to everyone else
• Roll-out team — This consists of product and engineering leadership (first wave stakeholders), relevant PMs and engineers, and your client success manager
• Timeline — The entire plan is a roadmap for execution and program management. Start two parallel work streams: (1) Technical Implementation, and (2) PM Enablement.
• Technical implementation milestones — Key milestones in technical implementation would be platform installation, training about the API and data structures, sample instrumentation, testing, and full-on instrumentation.
• PM enablement milestones — Key milestones include taxonomy training, platform training, taxonomy creation, sample analyses, and ongoing analytics.
• Scaling plan — Here you will define a roadmap of future products or features to instrument.

Your vendor should be providing support. The client success manager will facilitate the roll-out. The CSM will also have the resources, templates, and approximate timelines to smooth this process.

Larger organizations may benefit from project management support on this. The technical implementation and PM enablement work streams can run in parallel until the taxonomy and product instrumentation happen. Instrumentation is dependent on a well-defined taxonomy.

“How long will all this take?”

This is tricky to answer since the shape of your organization is likely different from mine. Alas, I’ll provide some reasonable expectations for the self-contained steps.

Timing Guidance

• Defining what you need: ~2–3 days
• Finding vendors: ~1–2 days
• Getting demos from vendors: ~1–2 weeks (depending on your schedule and how many vendors)
• Trial experiments and synthesis: ~2–4 weeks (depending on technical hurdles and how much data is coming in)
• Getting justification assets from vendors: ~1 week
• Planning cost scenarios: ~2–3 days
• Defining operational plan: ~1 week

That adds up to one to two months of calendar time. All the other pieces — navigating your organization, evangelizing, securing budget and procurement — largely depend on your company’s pace and complexity.

Conclusion

So… A few weeks after signing the agreement, you should be swimming in data. Take a deep breath. The fun part is just about to begin.