Short-term wins often disappear long-term

I would encourage everyone, if you can, look at some of the experiments that you thought were your biggest winners. Look at the downstream metrics for a year, two years on that experiment. And I'll bet you'd be surprised how many times the metric is different than what you thought it would be after a year.

I think there's probably two things that have been very common. And I would say in quite a few cases, you get a lift on a metric up front, a more short-term metric... And then you look a year later, and there's actually no incremental lift on GMV from that cohort.

The percentage refers to how often growth experiments that show positive short-term results turn out to have no long-term impact when measured after a year.

It's in the 30 to 40% range.

Most of growth loops spin out their ability to produce meaningful results for you within the first five to six to seven years.

OEC stands for "Overall Evaluation Criterion," Kohavi's framework for defining metrics that A/B tests should optimize for.

To me, the key word is lifetime value, which is you have to define the OEC such that it is causally predictive of the lifetime value of the user.