Skip to content

Home / Journal / AI Development

AI Development28 July 20266 min read

Write the eval before you write the prompt

Every AI feature that quietly rotted in production had the same origin story: a great demo and no way to tell whether it still worked on Tuesday.

A demo proves a thing can happen once. An eval proves it keeps happening. The gap between those two sentences is where most AI projects die, usually two months after the launch announcement, when nobody can say whether the output got worse or the users got pickier.

So we invert the order. Before a prompt is written, we collect thirty to fifty real cases from the client — the actual messy inputs, including the ones that made someone escalate to a manager. Each case gets an expected outcome, graded by whoever owns the process today. That set becomes the contract.

Once the harness exists, prompt work stops being taste and starts being measurement. You change one instruction, you re-run, you see 0.81 become 0.88, and you keep it. You see it become 0.74, and you throw it away without arguing about it in a meeting.

The second benefit is slower and more valuable: the eval set is the only artefact that survives a model change. When the underlying model is swapped for something cheaper or newer, the set tells you in ten minutes whether the swap was safe. Without it, you are shipping a rumour.


← All notesNext: The automation audit: find the hours before you write the code

Next step

This is how we would build yours. Talk to us.