Qualitative insights complete quantitative testing

AB tests are great, but the AB test rarely tells you why it changed in the way it did. You don't have to guess. I know somebody who can get you an answer.

We changed seven characters and made Airbnb millions of dollars, because what we found out was really simple. The button feels scary. The CTA on the button feels scary.

Being able to identify the various micro and macro loops, how they're all connected, being able to document them in a qualitative model to communicate a shared understanding of how you grow, it's really powerful. Augmenting that then with the quantitative side of things that helps guide quarter to quarter focus and ensure you can be intentional about where you're investing, that becomes a big enabler.

Error analysis refers to systematically examining AI system failures to identify patterns and root causes, which the speakers demonstrated earlier in their presentation.

I think a lot of people prematurely do A-B tests, because they've never done any error analysis in the first place. If you're going to do A-B tests and they're powered by actual error analysis as we've shown today, then that's great, go do it. But if you're just going to do them based on what you hypothetically think is what is important, then I would encourage people to go and rethink that.