Every time I hand an agent a job it has done before, it starts from zero.
It reads the input, plans, calls its tools, checks itself and writes the answer. The tenth run costs as much as the first, and most of what happens in the tenth run already happened in the first.
The idea I posted was simple. I don't want to change the model. I want a wrapper harness that watches the LLM do a task like "analyze my profile", learns the process, and turns it into a program. The next profile should be much faster, and the decisions left for the model should be smaller than the first time.
That is using an LLM as a compiler instead of a runtime. I asked Fable 5.2 in Claude Code to test it properly on my own machine: public datasets with human labels, real local models, and training, calibration and test data kept apart. It designed nine experiments, ran them on an M4 and stored 18,529 real model answers along the way. The code, results and full report are open.
A program is a ladder of cheaper answers
Compiling does not remove the model. It moves it. Arithmetic, lookups and formatting become code. A judgment such as "which department does this request belong to" becomes a typed decision, answered by the cheapest tier that is sure: an exact cache, mined rules, a small decision model, then one narrow LLM call, and finally the full agent.
On my M4 a local decision takes between 0.2 and 9 milliseconds. One typed call to a local 4B model takes 267 ms, and only because the task contract sits in a cached prompt prefix. Re-reading the contract on every call takes 6.3 seconds.
How much the fast tiers can take over depends on the task. With a 2% budget for silent errors, a small embedding model could answer 94% of one assistant dataset and 76% of a banking dataset. On a third dataset no model could be certified at 2% at all. A compile step has to be allowed to say: this decision stays with the model.
Confidence is not familiarity
The dangerous failure showed up when I held out whole groups of requests. The fast model was trained on some intents, then tested on intents it had never seen, with the same set of possible answers.
Calibrated the usual way, on a random split, it kept its 2% promise on familiar groups. On the never-seen groups it answered 76% of requests and 23% of those answers were wrong. That happened in 20 of 20 random splits. 68% of those mistakes were made with a confidence of 0.9 or higher.
The fix is to calibrate against the shift you expect. Hold whole groups out, test them separately, and add a familiarity score that does not depend on the model's confidence. I used the distance to the nearest training example. That cut wrong answers on unseen groups from 17.6% to 1.6% of requests. It also cut the share of familiar requests the fast tier answers from 95% to 63%, and it still went over budget in 35% of the splits, because rows from one group fail together. It is an estimate, not a guarantee.
Typing noise showed the same problem from another side. With 15% of letters mistyped, the embedding model was 51% accurate and still answered 53% of requests, 24.5% of them wrong. A plain character n-gram model answered 43% and stayed at 2.3% wrong. Its confidence dropped with its accuracy.
A verifier has to fail differently
A fast answer should not be its own verifier. But a second model trained on the same features is almost the same model.
When the fast model was wrong on a never-seen group, an ensemble of five networks on the same embedding gave the same wrong answer 70% of the time. A character n-gram model did so 38% of the time, and a local LLM 32%. The n-gram check costs 0.2 ms and catches 62% of those errors, so it became the default verifier.
The local LLM was not a safe fallback either. Qwen3-4B was right on 95% of familiar requests and 78% of unseen-group requests. It took its "none of these" exit for only 45% of out-of-scope queries, and it gave an in-scope answer to 10 of 12 prompt-injection strings.
Putting the guarded fast tier in front of it gave the same quality, 11.4% wrong against 12.4% for always asking the LLM, with 0.46 model calls per request instead of one. With the human label as the fallback instead of the LLM, the same fast tier delivered 1.0% wrong answers and escalated 46% of requests.
Programs rot
I compiled a headline router on one year of HuffPost headlines and let it run on the next five. In 2013 it was wrong on 11% of what it answered. In 2018 it was wrong on 65%, and it was still answering. By then 53% of the headlines belonged to sections that did not exist when it was compiled.
Then a scripted loop over 2,790 held-out requests with four kinds of drift. Anything the program refused went to the human label. The guarded program handled new groups, a burst of out-of-scope requests and noisy input with 1.3%, 2.7% and 1.0% wrong answers. The naive one was wrong on 16% of the whole stream.
The fourth drift was a policy change: requests about physical cards moved from one department to another. The inputs looked familiar and the program was confident, so 81% of the re-routed requests were answered under the old policy. Re-checking 10% of the fast answers did not catch it in the 640 requests that followed, and re-checking 40% took 200. Announcing the change, and switching off the affected answer class at once, brought it to 0%.
Two rules came out of this. A compiled answer has to carry the version of the contract it was compiled for. And a program must never retrain on its own unverified answers: that was the worst policy in the loop.
The wrapper harness, tested
This is the experiment closest to my original post. The agent analyzed public résumé profiles in three free-form calls: extract the facts, assess them, write a summary. The harness recorded 60 runs. For every output field it searched a small family of candidate programs, and admitted one only if it agreed with the agent on 40 other recorded runs.
On 80 profiles it had never seen, the compiled version took 3.8 seconds instead of 11.2. If the prose is still generated by the model, 6.7 seconds.
The decisions shrank more than they disappeared. Only 1 of 6 short fields became pure code. The rest stayed with the model as one narrow, typed call that reads 528 tokens instead of 1,885 and writes 78 instead of 182.
The recordings also audited the agent. In 44 of 100 runs the profile text showed no education at all, and the agent still reported a degree in 33 of them. A rule that copied this habit agreed with the agent on 70% of runs. The best honest rule agreed on 45%.
It went the other way too. A rule written from the task description matched the human label on 83% of the profiles it answered. The agent matched on 67% of the same profiles. Judged by agreement with the agent, that rule was rejected.
So the trace is evidence, not the specification. Compile from the contract, and check against reviewed outputs wherever you have them.
The same test with a frontier model
Everything above used a 4B model on my laptop as the LLM tier. So I gave the same contract and 1,949 of the same held-out requests to Claude Opus 5.5 at medium effort. I had no API key on this machine, so it answered in blind batches through Claude Code subagents. That means real answers, but no latency and no cost figures.
It was a different tier. On never-seen groups it was right on 95% of requests against 78%, with 1.3% wrong answers against 20%. It took the "none of these" exit for 96% of out-of-scope queries against 47%. It gave an in-scope answer to 1 of 12 prompt-injection strings, where the local model did for 11. And when the fast model was wrong on a never-seen group, it repeated the same wrong answer 4% of the time, against 33%.
That flips one conclusion. With the local LLM, the guarded fast tier cost nothing in quality. With Opus, asking the model every time was 1.0% wrong, and the guarded fast tier in front of it was 1.8% wrong while saving 55% of the calls. The fast tier's own wrong answers were 0.9% of all requests. A naive fast tier in front of the same model was 9.2% wrong. The better the fallback, the more a careless compile costs.
What this does not show
Outside that last section the LLMs are 4-bit local models of about 4B parameters. The frontier model ran on one frozen split, in batches, without latency or cost. The last tier in these tests is the human label standing in for an agent. The harness was tested on one workflow, with candidate programs written by the coding agent after reading the recorded runs. And none of this touches perceptual work like video to Blender, where the verifier is the hard part.
I like where it landed. The idea survives with less magic than I hoped and more structure: a ladder of cheap answers, a gate that knows what it has not seen, a checker that is built differently, and a contract that outranks the trace.
Correction, 11 November. The first version of this post said 65% of the re-routed requests were answered under the old policy, and that re-checking 10% of the fast answers noticed after 400 requests. I had counted applications for a new credit card as re-routed, and the policy text does not cover them. The naive program's total was given as 21%. With only the request types the policy names, the numbers are the ones above, and the production-loop figure and the film were redrawn.
