RATCHET: iterate your agent against a frozen yardstick
If you can't trust a model that writes its own verdict, freeze the yardstick. The RATCHET pattern: keep-if-better, else revert — with 0.83 Spearman proof.
In the earlier posts in this series I took apart two ways an agent lies to itself: model self-evaluation is a feedback loop of self-deception, and multi-agent consensus is just the same mistakes sung back to you in chorus. If you can't trust a judge who writes his own verdict, you need something the model can't touch: a frozen yardstick.
This is the first positive post in the series — less complaining, more of a pattern that actually works for me. I call it RATCHET. In agentic engineering it means one thing: you measure every iteration — the prompt, the model, the agent code — against the same, frozen yardstick. An immutable yardstick. A reference point you set once and then don't move.
The mechanism is trivial
The whole trick fits into four steps. And there's no AI magic in any of them.
RATCHET — KEEP-IF-BETTER, ELSE REVERT
- Define a hard test. You write the yardstick BEFORE you iterate, not after. Yardstick first, change second.
- Ship the change. One prompt, one model, or one slice of agent code — ideally on its own.
- Run the test. Same test, same data, one number out.
- Keep-if-better, else revert. If the result isn't measurably better, you roll the whole thing back.
Notice the word "measurably." Not "feels better to me." The number is either higher, or you go back to the previous version. This is a regression test for quality — the same kind you already keep in code, except it measures meaning rather than syntax. Agent iteration stops being a matter of mood and becomes a ratchet: quality can climb or hold, but it never slides back quietly.
It sounds simple because it is. The difficulty isn't in the code, it's in the discipline — in not giving in to the urge to move the yardstick the moment it ruins your good mood. Because the moment the score drops is exactly when you most want to believe the test is wrong, not your change.
This is what I do hands-on — advising on AI strategy and building agents that survive the demo.
The spirit level you're not allowed to move
RATCHET works like a spirit level on a construction site. The trouble starts the second you start nudging it to match what you've already built.
"If you adjust the level halfway through building a wall so it fits the crookedness you produced yourself, you'll never build anything plumb."
That's exactly what most teams do with their quality evaluation. They iterate the agent, and when the new version scores worse, they "fix" the test instead so it shows green. After three rounds of that, the yardstick only measures what you wanted to see. Immutable means immutable: once you freeze the yardstick, its number is the only arbiter — even when it stings.
Proof from OffBall
I'm not writing this from the slides — I'm writing it from the trenches. On one of my projects, OffBall, I built a yardstick like this and tested it against people. The question was simple: does the automated scorer rate quality the same way someone who knows the domain would?
It scored almost exactly like live experts — it ranked the same things in the same order. Measured hard, that's 0.83 on Spearman's rank correlation. In other words: the yardstick's ranking and the experts' ranking are practically the same list. Reliable enough that the whitepaper built on that evaluation passed three rounds of review without a single substantive edit. No AI magic. Just a hard, repeatable measure the model can't game.
For me that was the moment evaluation stopped being a formality and became a tool. I could swap the model and the prompt knowing that if quality dropped, the yardstick would tell me loudly — before a reviewer or a client did.
Gut-feel iteration vs. a yardstick
Most teams iterate on gut feeling. They change the prompt, the model, and the tools all at once, run a handful of examples, and say "probably better." That isn't engineering. It's guessing with good PR. And two weeks later nobody remembers why the agent behaves the way it does. I'd rather know — know that the new version is a concrete X points better than the last one, and know exactly which change earned it.
- One variable at a time. Because the yardstick is fixed, you can attribute every move up or down to a specific change. Without that you can't tell whether the new model helped or the new prompt did.
- No room for self-deception. The yardstick doesn't know your intentions. A higher number or a revert — end of discussion.
- History as a ratchet. Every regression is visible and rejected, so your system doesn't quietly drift backward between one deploy and the next.
The simplest move you can make
Before you change anything in your agent, define one hard test and freeze it. Then iterate — but only keep-if-better, else revert. If the model can't game the yardstick, it can't fool itself either. The question worth asking today: what does your quality yardstick look like when nobody's watching?
