Count first, complicate later

Basic counting is the most powerful analytical technique in data science because it's so simple and it's kind of undervalued in many cases.

Signal not noise. So typically, whenever I hire people, I want to hire smart people and kind of get out of their way. But the biggest thing I want to focus on is what is your thought process when it comes to data?

"Before" refers to traditional data science approaches, and "the confusion" refers to the debate over whether AI products need specialized evaluation methods versus standard data analysis.

This is the same data science as before, and I think that's what's causing the confusion is, 'Hey, we need data science thinking,' and AI product is helpful to have that thinking in AI products like it is in any product is my take on that.

"This agreement" refers to measuring how often an AI judge system agrees with human evaluators when assessing AI model outputs.

A lot of people go straight to this agreement. They say, 'Okay, my judge agrees with the human some percentage of the time.' That sounds appealing, but it's a very dangerous metric to use, because a lot of times, errors, they only happen on the long tail and they don't happen as frequently.