"Evals" refers to evaluation systems used to test and measure AI model performance in product applications.
To build great AI products, you need to be really good at building evals. It's the highest ROI activity you can engage in.
Error analysis is where the magic happens
Craft → Product Sense
"This" refers to building evaluations (evals) - systematic tests to measure AI application performance and quality.
Everyone that does this immediately gets addicted to it. When you're building an AI application, you just learn a lot.
"The same exact process" refers to error analysis - systematically reviewing AI application outputs to identify problems. "Annotating things" means labeling data examples as correct or incorrect.
Put your product hat on and get into, is this really good? That's where the fun part is. You're looking at data. It's like, okay, you're annotating things. Actually, I was just looking at a client's data yesterday, the same exact process. It's a lot of fun, actually.
Put your product hat on and get into, is this really good? That's where the fun part is.