Don't become the oracle

Building software with AI is fun. Often a few rough instructions are enough to get remarkable results. But this lightness is fragile. As soon as the scope of your project grows, you’ll find yourself doing more and more validation.

The pattern is always the same. You point out that something is wrong and describe how it should be instead. The AI thanks you, fixes it, and hands back a new result. You look again, but now something else is off, and on top of that it quietly changed a part that was already fine. So you correct that too. And round it goes.

The issue is that you’re the oracle. You’re the one who decides whether a result is correct, and only you know what that looks like. The AI doesn’t. It can’t judge its own progress or tell whether something is actually right. And as the scope grows, so does the amount of validation, until the speed of the whole project is bottlenecked by your own painfully slow ability to check the AI’s output. Ugh.

The cool thing is that for some problems the oracle already exists. Say you’re building an HTML5 parser. There’s a well-maintained test suite that defines the correct output for any given input, which means the agent can check its own work and actually know when it’s making progress.¹

Bun’s recent rewrite from Zig to Rust is a great example of this in the wild. Its existing test suite already captured how Bun was supposed to behave, so the agents could translate hundreds of thousands of lines and keep measuring themselves against it until everything passed.²

The takeaway for myself, and maybe for you too, is this. Before you start building your next application, think about the oracle first. If you don’t define it up front, you’ll become it and find yourself in manual validation hell. Maybe one already exists out there that you can reuse. If not, invest the time and build one yourself. That’s at least how I’ll approach future AI engineering projects.