Experimentation was never historically in science about winners and losers. Experimentation is always very hypothesis driven. It's about, what are you learning?
Marketplace lessons from Uber, Airbnb, Bumble, and more
November 09, 2023
Featuring: Ramesh Johari (Professor, Stanford University)
8 quotes · 5 insights
Watch Full EpisodeLearning beats winning when experimenting
Learning isn't free. When you don't know something, it seems not even a question that you would allocate some of your samples to all options. After the fact you're like, 'Treatment was better. What the heck were we thinking? Why'd we give all those samples to control?'
Platform thinking unlocks exponential growth
A marketplace business never starts as a marketplace business, because what we think of as a marketplace business is something which at scale is removing the friction of the two sides finding each other. But when you start, you don't have that scale.
Maybe we shouldn't talk about the concept of a marketplace founder. Really there's founders. Every founder is a marketplace founder. It'll be a choice they make after they grow as to whether they want to build a platform.
A/B testing has fundamental limitations
You can't experiment your way out of everything. What people decide to test in a world that has promoted experimentation for everything tends to be more incremental by design.
AI amplifies judgment, doesn't replace it
What AI has done for us is it's massively expanded the frontier of things we could think about our problem, hypotheses we could have, maybe things we could test. I really think actually what that does is puts more pressure on the human, not less.
Data tells you what, not why
Prediction is inherently about correlation. But when we ask people to make decisions, we're asking them to think about causation. The distinction between causation and correlation matters a lot.
What AI has done for us is it's massively expanded the frontier of things we could think about our problem, hypotheses we could have, maybe things we could test. I really think actually what that does is puts more pressure on the human, not less.