Five Lessons on Continuous Discovery
The antithesis of modern Product is mindlessly building a list of features. It’s an anti-pattern to rarely talk to your customers. It’s a trap to build for months and then release without active feedback.
You can read this article for free at www.austinjnichols.com/5-lessons-on-continuous-discovery
So why then do we see these habits still creep into our daily lives as product teams? Every team is different, but you’d be lying if you haven’t fallen into one of these traps at some point. Maybe it’s because it’s easier said than done. Maybe it’s because teams don’t have a true playbook to execute against.
Well — throw out the excuses.
Teresa Torres’ book, Continuous Discovery Habits, is a master class on building a healthy product culture that will result in great products and a growing business. She defines continuous discovery as:
Weekly touchpoints with customers
By the team building the product
Where they conduct small research activities
In pursuit of a desired outcome.
I highly recommend you read the full book, but here are five key takeaways that you can start using today.
1. Use small experiments to move fast
I read Inspired by Marty Cagan early in my PM career. Among the many lessons was this imposter syndrome inducing quote,
Teams competent in modern discovery techniques can generally test on the order of 10–20 iterations per week. — Marty Cagan
How in the world are we supposed to do that much discovery? How do we avoid “big-event discovery” that takes weeks? The answer is in breaking down experiments into centering experiments around assumptions, not solutions.
Assumption testing is generally quicker than idea testing, and the faster pace helps us to guard against the escalation of commitment. The less time we invest in an idea, the less likely we are to fall in love with it. — Teresa Torres
It might sound intimidating, but it’s a hell of a lot effective than building a “Proof of concept” for 2 months before you can start learning.
Here’s how to do this:
- Generate sets of ideas instead of a single solution for the opportunity you’re evaluating. Find the common assumptions between ideas. Design experiments that address the most important.
- Simulate experiences that allow us to test behavior that validates the assumption. Don’t simulate more than you need. Use approaches like unmoderated user testing and one-question surveys.
- Define success criteria up front to align the team before results start coming in. Use numbers instead of percentages to not game small sample sizes.
- Don’t get hung about being too scientific. While we do want rigor, we aren’t uncovering core truths about the world. Product discovery is as much art as it is science.
2. How to Prioritize Assumptions
But we have so many assumptions! How do we know where to start? You’ve probably heard of the term RAT (Riskiest Assumption to Test). But Ratatouille only gets us headed in the right direction if we can accurately assess what is most important. Teresa gives us a framework to help.
First, use story mapping to help identify hidden assumptions. Make sure you understand the types of assumptions you have by labeling them as desirability, viability, feasibility, usability, or ethical. This helps you identify blind spots before you actually start the prioritization process.
Next, get pumped to use ol’ reliable — a 2x2 matrix. We want to use assumption mapping to find “Leap of Faith Assumptions.” This boils down to mapping our assumptions based on how critical they are to our achieving our opportunity AND how confident we are in the assumption.
Teresa tells us not to worry about precision too much here,
You are mapping assumptions relative to each other. For example, once you place the first assumption, for the second assumption, you can now ask, “Do we have stronger or weaker evidence for this assumption as compared to the first assumption, and is this assumption more or less important than the first assumption?” and place it relative to the first assumption. — Teresa Torres
Once we have our matrix filled out, it should be clear to the team where we need to spend our discovery efforts for the coming weeks.
3. Use the Right Types of Outcomes
PMs might as well have Outcomes over Outputs tattooed on our bodies for how much we talk about it.
But it’s not uncommon for teams to struggle with setting the right north start metrics. Torres gives us a helpful framework by differentiating outcome types.
- Business Outcomes | These are about the health of the business — revenue, churn, ACV, etc.
- Product Outcomes | These evaluate if our product is actually driving the business value we want. They should ladder up — customer behavior, NPS, PMF scores, etc.
- Traction Metrics | These evaluate specific usage of specific features — video views, reports created, tweets sent with media, etc.
Product teams are most empowered and successful when they are rallying around product outcomes. There are too many factors outside of the team’s control to own business outcomes and they are most often lagging indicators. Traction metrics are often too specific but can be helpful inputs into our product metric.
This framing is one of the missing pieces to unlock your teams to be truly empowered and effective.
4. Avoid Speculation in Interviews
The cardinal sin of user interviews is to ask leading questions. But avoiding those isn’t enough. In fact, there’s an even bigger boogeyman that can tank your research — who, what, why, where questions.
[These types of questions]…require that we recall facts without context. This process is prone to cognitive biases — common patterns in mental errors that result from the way our brains process information. We are bad at quantifying how often we do something. We often speculate about what we did, when, and why. We tend to favor generalities over specifics. We give answers that are influenced more by our sense of identity rather than our actual behavior. And we tend to come up with coherent reasons to explain our behavior that are often not grounded in reality. — Teresa Torres
This might sound harsh, but it’s just the way our brain works. Instead, Torres encourages us to use story-based interviewing. First, ensure you have your key research questions defined. Then translate those into story questions.
Instead of asking, “What causes you to watch video on our platform for longer?” — ask, “Tell me about the last time you watched a video longer than 10 minutes on our platform.” If they’re not using your product in that way yet, you can go even broader with something like, “Tell me about your last binge-watch session.”
The key here is to ask the user to be as specific and descriptive as possible. The more details about the story, the more insight for your team. This will give you more reliable data about your assumptions. This data is the life-blood of continuous learning.
5. Show your work. Always.
Have you ever gotten up at a product review and walked through your well-thought-out slides only to get back confusion, disagreement, or crickets? We’ve all been there. But what can we expect when we try to jam all that insight, progress, and meaning into a few nice bullet points?
Torres encourages us to be visual in everything we do — especially when communicating with stakeholders.
Here’s how you can do this:
- Don’t focus just on conclusions (the roadmap, the solution, or a backlog). If this all we share, we’re setting ourselves up for an opinion battle that lacks context.
- Instead, show folks how we got here. Walk them through the opportunity solution tree. Walk the lines with them. This is how you as a team built confidence — it should do the same for your stakeholders.
- Even better — bring them along for the journey. By sharing our discovery frameworks visually with the team, we are inviting them to co-create. This will improve buy-in and make it easier to show progress along the way.
- Find the right level of detail. The Group PM above you is usually going to want a lot more detail than your CTO. Experiment with how deep you go into the frameworks and find the balance between setting context without overwhelming.
What about you?
If you’ve read the book, what are your key takeaways? What are you planning on implementing first with your teams? What’s been the hardest part of building the Continous Discovery muscle? Let me know in the comments or on Twitter @nichols_austinj.
You can read this article for free at www.austinjnichols.com/5-lessons-on-continuous-discovery.
