Making a Good Bet: Building Products with Incomplete Information
Do we really need evidence, or can we YOLO our way to success?
I saw a Twitter/X post recently from my buddy
(more from him later) that made me have a bit of a think:“Even if something is not quantifiable, and the impact is unknowable, does not mean it is not the most important thing right now. This is a crucial lesson for prioritization. There are many good PMs and product leaders out there who forget this.“
This post made me think for a couple of reasons. The first reason it made me think is that I broadly agree with the central point.
The second reason this post made me think is that this statement could be used to justify doing literally anything.
Let's dig into it. But, as you grab your shovel, do make sure to subscribe if you’re not already!
When Product Managers Stop Making Progress
I’ve seen a few situations in my career where product teams seemed caught in analysis paralysis, unable to make a meaningful move, circling the airport but forgetting to land. This leads to escalating tension between product managers and business stakeholders as well as probably dissatisfied customers. The product management team is seen as “the place where good ideas go to die”.
A secondary effect is that, eventually, the CEO or some other leader ends up HIPPOing a solution onto the roadmap and everyone complains (but is secretly quite glad that they know what to work on now).
Sometimes you’ll have great information and can proceed with confidence. Sometimes you will indeed need to take a punt on something with unclear chances of success. But what stops product managers from doing this?
Lack of Product Strategy
It’s hard to know where to place a bet if you don’t know what you’re aiming for or how you’re trying to get there. It’s not exactly a surprise that many companies don’t have a strategy at all, let alone a product strategy. Product managers know that they should be doing something to get away from randomised feature requests and top-down feature edicts, but they don’t know what. In some places, they’re actively prevented from doing so.
No Usable Data
In an ideal world, product managers should have data at their fingertips to enable fast decision-making. It’s a lot easier to decide based on data if you don’t have to go and get the data before making a decision. This data could be quantitative usage data, market research, customer research, competitor analyses or any number of other potential inputs. If you don’t have it, you can’t move forward with confidence, and if you don’t have confidence, maybe you can’t move forward at all.
Past Product Trauma
Part of this lack of confidence may simply boil down to the fact that the product management team tried to do something before and it went horribly wrong. They built a big feature on an educated guess, spent months building it, released it and had nothing to show for it. Often, in situations like this, the product team are the ones left carrying the can.
Fear of Repercussions
In psychologically unsafe organisations, this can lead to blame and repercussions. We often hear of learning organisations and being able to take risks, but there are many organisations out there that seem to have zero risk tolerance. Everything has to be a slam dunk. If you didn’t get it right last time, you’re going to get judged, blamed and maybe even fired.
Product Management Purism
Or, maybe your product managers are so addicted to all of the classic product management books that they simply don’t believe they can do anything without following the recipes inside. This is an interesting one because I’m pretty sure most of those authors would not advise being so dogmatic in all circumstances. However, particularly in low-functioning product cultures, it can be incredibly comforting to fall back on what people think are best practices.
Undertrained Product Managers
I’ll choose my words carefully here because I don't believe in “bad” product managers per se, but there are a wide range of skills, mindsets and appetites across product management teams. This is, of course, exactly the same for any other profession too! In situations like this, product leaders have a strong responsibility to coach and mentor their teams and help with better decision-making.
Making Progress Responsibly
As mentioned earlier, the problem with “Even if something is not quantifiable, and the impact is unknowable, does not mean it is not the most important thing right now” is that it can be used to justify literally anything.
All product managers have horror stories about that one feature they spent too much time on that didn’t make a difference, the CEO’s vanity project that didn’t pay off, or the speculative integration that would unlock an entire market segment that unlocked precisely zero market segments.
The issue here isn’t that we shouldn’t think big or overanalyse. I generally recommend that product managers start Thinking in Bets. There’s no binary "good idea” versus “bad idea”, just a probability of success. This probability could start low or start high. But it’s sure easier to put all your money in the pot if you’ve got a good starting hand.
The difference between a world-changing invention and a dud vanity project is, at the beginning at least, zero. You can win a game of Texas Hold’em with any two starting cards, but most people don’t go all-in on a 2/7 offsuit.
Here are some things to try:
Try the sniff test
There’s always room for gut feel in product management. Ultimately, does an idea make sense on its face? Based on what you know about your product, market and customers, is this thing stupid or not? Discuss it with your colleagues! You can also happily use ChatGPT, or your LLM of choice, to bounce some ideas back and forth as a kind of DAaaS (Devil’s Advocate as a Service). I wouldn’t bet a billion dollars on a sniff test, but I would be a lot more confident proceeding further with something if it just feels right. There is, of course, a danger here… some people are just massively overconfident in everything. There’s a lot of unconscious bias that could creep in here. But still do the work.
Identify assumptions
Whilst thinking this through, you should be furiously writing stuff down. One important thing to write down is a list of assumptions that must be true for this to be a good idea. There are many good ideas that have a load-bearing assumption that, if not true, would make everything collapse. Look at these assumptions. Which are the riskiest assumptions, the things that must be true, and which are kind of irrelevant to the success of your initiative? How can you test them?
Use what data you do have
If you have any data at all, look at it and try to find anything that proves or disproves your hypothesis. Again, this could be quantitative data, customer feedback, discovery interviews, industry reports, requests from the sales team, or anecdotes from a meetup you went to. You need to weigh the data accordingly and not treat absolutely everything as gospel, but anything is better than nothing. You’re looking for signals.
Build the smallest thing you can (so you know whether you should build a bigger thing)
You may still find yourself with very limited information, and no real justification to build whatever you’re thinking of building. Maybe the data’s just not available. Maybe the thing you’re thinking of doing is so cutting-edge that it’s impossible to know until you see it. That’s OK, you can still make a bet but the crucial element here is the size of the bet. Putting a billion dollars and 12 months of engineering time into something is a whole lot more palatable if you know it’s going to be a guaranteed success. But nothing’s guaranteed success, so start small. Run experiments. Get prototypes in front of people. Build something, but just not everything. Test the water with a smaller bet so you can make a bigger bet with confidence.
Avoid the sunk cost fallacy
As with Texas Hold’em, you need to know when to go all-in and when to fold the hand. You can’t win them all, and it’s lunacy to throw good money after bad. In the past, I’ve seen product and engineering teams get disappointed that “our hard work is being thrown away, what a waste of time!”, but this is only true if you didn’t learn something along the way. Building a small thing to see if you should build a big thing only works if you are prepared to not build the big thing.
And, besides, there’s always another Next Big Thing to sink your teeth into!
Overall, as with everything in product management, it depends. Good product management is as much an art as a science, and there’s no formula that guarantees success. As product managers, it’s our job to make forward progress for our customers and our companies, not just sit around looking at the floor. It’s important to be comfortable with ambiguity, react to new information and make bets from time to time. Just make sure you bet responsibly and know when to fold.
One Knight in Product - Hot Takes edition
I’ve released my first “Hot Takes” episode with returning guest Debbie Levitt. You may remember her from a few weeks ago when she was explaining how Continuous Discovery is destroying the craft of UX Research. Now she’s back to talk about her fears that, if we’re all doing each others’ jobs, we’re just making it easier for AI to replace us sooner.
You can check the episode out audio-only on your favourite podcast app or grab it here. Alternatively, you can check the episode out on YouTube via the embed below.
If you would like to have a chat about your hot take, head over and grab an interview slot. I’d love to chat about whatever’s on your mind and get you in front of my audience!
Deep Dive into Fractional Product Leadership
I wrote my own article on this topic in this very newsletter, but
recently reached out and asked if I’d like to collaborate on one for his much larger newsletter. Unlike my article, we interviewed a collection of fractional leaders from around the world and synthesised it all into one mega-article. Some of it’s free, and some of it requires a subscription, but do check it out if you are interested in fractional work!
My appearance on the Go-To-Market Strategist Podcast
Speaking of fractional work, I also recently had the chance to speak to my friend and former podcast guest
on her very own podcast. You may remember Maja as the author of “Go-To-Market Strategist” the book. We spoke about moving from full-time product leader to being a successful consultant, and a lot more besides. Check it out on your favourite podcast app, get it here or at the Apple podcasts link below.Thanks for reading
I checked my “Pocket Saves” and found out I haven’t really bookmarked anything over the last week, so you’re stuck with my content alone. If you actually enjoy being stuck with my content, here are some ways you can keep in touch:
Join my Virtual Networking weekly calls
Join my real-life London meet-ups
If you have any comments, questions or complaints about this newsletter, please feel free to reach out! The same goes for any topics you’d like covered in future issues.
I’m still averse to charging for Substack subscriptions but, if you find my work valuable, please consider a one-off or recurring donation.
Thanks!
Post product trauma is so real. My post next Wednesday I’m going to dig into that topic a bit. Great post!
Thanks so much for being my podcast guest and sharing your consulting/advisory wisdom 😇 I learned a lot from you!