Crystal Widjaja

Do not treat metric gathering as entertainment. It's not there for you to be like, 'Oh, that's interesting, how novel,' and then not act on it. So real news is information that changes what you do in the real world.

"This" refers to insights derived from data measurements - understanding the "why" behind user behaviors rather than just observing what happened.

The insight will provide value when you have this, why answered? Why is this person doing this thing? Here's why. And then you are going to act differently.

So we almost create this artificial friction to help differentiate how deeply a user wants something or needs something. And if the user doesn't fill out that questionnaire, maybe they're actually looking for something else.

"Made to Stick" is a book about making ideas memorable; the "sandbox" structure refers to adding constraints to make thinking more specific and concrete.

When you think about retention, that's just not specific enough. So there is this mental model that I use from made to stick where they'll tell you like, 'Lenny, think of everything in the world that is orange.' And you're like, 'An orange. What else?' And then if you change that structure with sandbox to think of everything orange that's in a construction site, then you really start to realize and grasp at concrete concepts.

GoFood is Gojek's food delivery service (like DoorDash or Uber Eats).

We're always looking for what is a specific reason that this user might have converted? For things like GoFood it would be things like when does a user try a new merchant if what people are ordering right now or just food that they already trust and know.

It's usually the step right before conversion. So if they aren't sure why the user opens the app or they aren't sure why the user got to this checkout page, it's often some copy or the path has been ineffective in some way.

So I talked about using drivers to sell GoPay. Before that, one thing that we did was to actually take someone's virtual account number and put it onto a picture of a credit card. You know what a credit card is, that's familiar to you. A lot of people didn't know what a digital wallet was.

We actually looked at the food that their friends had purchased and used that as a data set of, 'Hey, here's food that Lenny purchased and liked. Maybe you would like it too.' And so that was one way to hack the trust factor.

Crystal is discussing weekly retention benchmarks - the percentage of users who return in their second week after signing up.

So assuming that the frequency is correct, so you have a weekly frequency, if users are coming back, if it's a free product, 60%. It has to be at least 60%. If it's a paid product, I usually look at that more as maybe 20 to 30%.

The "80%" refers to week-one user retention rates for early-stage startups testing their product with friends and family.

If you are much smaller, your friends and family that better be near close to 80% no matter what, because if you can't even convince the people who care about you to use the product, it probably isn't going to solve the job for anyone else.

When we looked at the cancellation reasons and we saw that their number one reason was, I still have too much fear, we actually decided, well, let's just add a pause button then. Because canceling the subscription is a permanent solution to having too much fear.

I do find if you have a lot of people landing on a webpage or an app and then not doing anything, then it's probably copy. They haven't even experienced the product, it's clearly not the product that's wrong.

Even if you have a sample size of 30, the data you get back, generally, does not change but its precision will. So mathematically speaking, you're going to get the same level of trends, but the precision at which you understand those trends will become more deep if you have more data.

"That scale" refers to running experiments with small sample sizes of around 30 people, as discussed in the previous exchange.

Every idea is so cheap at that scale. You could do things that don't scale dramatically better with 30 people than at 100 if you're testing.

"Physics" refers to the fundamental constraints and parameters of your business model - things like your market, product capabilities, distribution channels, and monetization structure.

Step one is, what are the physics? Step two is when you think about loops and growth funnels and the quantitative inputs to each loop, does that fit into these physics or do you have to change four or five different things?

All of these characteristics of the experience and the context that can help you look at hey, when a user only sees two drivers on the screen, they're much less likely to convert than a user who sees five drivers on a screen.

The symptom of a bad data tracking approach is you have a ton of rows with a ton of events, but every event has one property or no property being tracked.

You wouldn't believe how great of a salesperson someone can be when you were literally trapped in a car with them going somewhere. And so you have this captive audience, captive attention, you have someone who has the incentive to cross pay or cross sell someone into GoPay.