Teach AI like junior engineers, not magic tools

Devin is an AI software engineering agent built by Cognition Labs that can write and test code autonomously.

The biggest thing I would say is it really is just treat Devin like your new junior engineer. I think folks come in and they see the blank page and they think of all sorts of various things that they want to try out. But a lot of it is just like, 'Yeah, let's figure out what tickets we want to get done today or this week and let's have Devin get started on those and let's start with the easier ones and then work with Devin and understand what things Devin needs to get set up to be able to test its own code and do this well. And then let's scale up over time.'

Devin is an AI software engineering agent, and "what you just saw" refers to a live demo where Devin added a newsletter link to their website.

You want to be giving Devin tasks, not problems. And a lot of these things like what you just saw, which was kind of like a quick front-end feature request or a bug fix or adding testing and documentation or things like that. One of the things that makes a loop really nice obviously is a quick way to iterate and test.

"This thing" refers to LLMs (Large Language Models), which Komoroske describes as "squishy computers" that don't do exactly what you tell them.

If it punches in the face, that's not a viable product. And so how do you design your products assuming that this thing will be squishy and not fully accurate and fully work?

Figuring out how to boil it down so that an agent can really understand what success and failure looks like is a lot of the game.

"That" refers to the DORA framework's four key metrics mentioned earlier. "Pipeline" refers to software development and deployment pipelines.

We can't just use that blindly now when we're using AI, as an example, because we have feedback loops much earlier and not even just at the local build and test phase. We have feedback loops throughout, and even sometimes in the middle of some of the pipeline.

We can't just put in a command and guess something back and accept it. We really need to evaluate it. Are we seeing hallucinations? What's the reliability? Does it meet the style that we would typically write?