Agentic Developer: Zero to Advanced, Part 8: Tool Design

Plain code, a small typed model, or a language model? Three kinds of tool, measured head to head on two real calls from the Part 7 pipeline.

agentic-ai
python
computer-vision
evaluation
Author

Hasan Goni

Published

October 17, 2026

Modified

October 18, 2026

Tip

This post is part of the Agentic Developer: Zero to Advanced series. It follows on from Part 7 – the inspection pipeline, and it is the fourth post of Arc 2, the agent loop.

Note

Rewritten on 2026-10-18 on the synthetic void dataset, with fresh runs of every experiment and rebuilt animations. The first version used GDXray images that I may not redistribute. Several findings changed, and I say which ones where they appear. Two stories from the first run (the token-probability bug and the Laya CPU fallback) happened on that data, and I kept them because the lessons still hold.

Part 7 ended with a result I did not expect: asked to classify boxes, the language model added nothing over a one-line contrast rule. That raises a design question this post tries to answer with measurements instead of opinions: for each step of a pipeline, what kind of tool should it be?

There are three candidates. Plain code (a rule). A small typed model, which reads text and answers a typed question (“yes or no”, “pick one of these”) with a probability, and never writes a word. And a language model. I put all three behind one interface and ran them on two real calls from the Part 7 pipeline, on the same held-out images as before.

One thing to say up front. The roadmap promised a head-to-head between TypeSafe’s hosted Jev and the open-source Laya. I could only run Laya. I never had a Jev API key, so the comparison here is a rule, Laya, and the local Qwen3-Coder. Jev’s adapter is written and not run, and the post says exactly where that leaves things.

Outputs are collapsed by default. Each one is the real captured output of the code above it.

0.1 Three kinds of tool

Every step in Part 7 was one of two kinds: plain Python (detect, measure, rules, decide), or a language model (classify, report). Here is the third kind, and what each one costs per call, measured on this machine (an NVIDIA GB10 shared with two vLLM servers):

Plain code Typed model (Laya) Language model (Qwen3-Coder-30B)
It gives you an exact answer a typed answer and a probability words, about anything
One call 40 ns 17 to 20 ms on the GPU, 523 ms on a CPU 198 to 564 ms for a one-number answer, 2.4 s to write a note
Tokens 0 0 79 to 171 per answer, 243 per note
Can write prose no no, it never generates text yes, and that is the only job only it can do

Daily-life version: a calculator, a clerk who fills in a form, and a writer. The calculator is instant and never wrong, but only does arithmetic. The clerk reads a page and ticks boxes, fast and cheap, but can only tick. The writer can say anything, which is exactly why you do not ask the writer to tick boxes you could have ticked by rule.

The three numbers that matter in practice are very different in size: nanoseconds, milliseconds, seconds. From a rule check to writing one note is a factor of about 61 million (40 ns against 2.4 s).

0.2 What a typed model is

Laya (Apache-2.0, pip install laya) takes a state (text or JSON) and typed questions, and returns typed answers with probabilities in one forward pass. Its question types are noul (is this statement true? gives a probability), choice (pick one option, gives a probability per option) and score. Because it only ever scores options, there is nothing to parse and nothing to hallucinate. One call:

notebooks/agentic-zero-to-advanced/typed_models.py
class LayaJudge:
    def noul(self, state, instruction) -> Result:
        ans, ms = self._predict(state, {"q": {"type": "noul", "instructions": instruction}})
        return Result(p=float(ans["q"]["noul"]), ms=ms, tokens=0)

Two things from its own README that I want to keep in view. The shipped checkpoints are weak zero-shot on its own benchmark (the base checkpoint scores 0.362 on typed decisions, below the majority-class baseline of 0.461), and the accuracy jump comes from fine-tuning. And its comparison numbers against Jev are third-party figures it says it never measured. I used the shipped checkpoints as they come, zero-shot, and I did not fine-tune.

0.3 One interface, so the experiment cannot cheat

If each judge is called its own way, the way it is called decides the result. So all of them sit behind the same two methods, noul and choice, and return a probability plus the latency and token count of that call. For the language model there are two ways to get a probability out of a text generator, and they are different judges:

  • “Number”: ask it to write {"probability": 0.8}. This is what Part 7 did.
  • “Token probability”: force a one-word answer and read the probability of yes against no straight from the model’s output.
notebooks/agentic-zero-to-advanced/typed_models.py
@staticmethod
def _token_probs(r) -> dict[str, float]:
    """Probability mass per normalised token. "Yes", " yes" and "yes" are the same answer, so their
    probabilities ADD UP (an earlier version kept only the last variant it saw)."""
    out: dict[str, float] = {}
    for t in r.choices[0].logprobs.content[0].top_logprobs:
        k = t.token.strip().lower()
        out[k] = out.get(k, 0.0) + math.exp(t.logprob)
    return out

That docstring is a scar. My first version of this function built a dict from the token list, so when the model’s top tokens included both Yes and yes, the smaller probability overwrote the larger one. The numbers looked plausible and were wrong. I noticed because I spot-checked individual answers: for a note that said, in so many words, “The inspection escalated because candidate 2 is uncertain”, the model gave P(escalated) = 0.02. After the fix, the same judge read that question at 95.8% instead of 57.9%. (That was in the first run, on the GDXray version, and it is why the fix is in the code.) Lesson: summary metrics will not tell you your harness is broken. Look at individual answers.

A second harness trap: Laya silently falls back to the CPU if putting the model on the GPU fails, with a warning. On this shared-memory machine it failed with a spurious “out of memory” unless CUDA was initialised first. My first latency numbers were therefore CPU numbers, hundreds of milliseconds. The adapter now refuses to continue if the model is not where I asked for it.

0.4 Experiment A: is this box a real defect?

This is the call that classify made in Part 7, now a pure judgment with an exact answer for every box: the detector produced 1,059 boxes on the first 200 dev images and the first 200 held-out images, 203 of them real. Same protocol as before: the dev images are where I pick any threshold (546 boxes, 122 real), the held-out images are where I report (513 boxes, 81 real, a base rate of 15.8%). I asked each judge the same question, once bare and once with the guidance from Part 7 written into the question (“brighter boxes are real more often: above contrast 30 about 7 in 10 …”).

I report a ranking score (AUC: the chance a random real box gets a higher probability than a random false alarm, 0.5 is a coin flip), plus calibration and an operating point, because accuracy alone is useless at a 16% base rate. I also bootstrapped the ranking scores over held-out images, because a difference of a few hundredths on 81 real boxes can be nothing.

Judge (held-out boxes) Ranking score, no guidance Ranking score, + guidance (95% interval) Brier skill (+ guidance) Calibration error (+ guidance) At the dev threshold: flagged / real One call Tokens
Rule (contrast at least 8) n/a 0.641 (0.59 to 0.68) n/a n/a 290 / 65 40 ns 0
Typed model, Laya 0.430 0.604 (0.54 to 0.68) +0.04 0.028 223 / 46 20 ms 0
Typed model, typed-decisions 0.457 0.478 +0.02 0.050 498 / 79 22 ms 0
Typed model, multilingual 0.307 0.378 -0.76 0.128 513 / 81 21 ms 0
LLM, writes a number 0.358 0.697 (0.64 to 0.76) +0.21 0.013 64 / 37 564 ms 171
LLM, token probability 0.671 0.643 (0.57 to 0.72) +0.08 0.126 252 / 52 216 ms 146

What I read from this, with the most solid points first:

  • Nobody beats the rule by a margin I can resolve. The best ranking score is the LLM writing a number (0.697), against 0.641 for a one-line rule. Paired over images, the difference is +0.056 with an interval of -0.014 to +0.132, which includes zero. Laya with guidance is -0.037 (-0.101 to +0.029), the token-probability LLM +0.002. With 81 real boxes I cannot rank these four. The rule costs 40 nanoseconds and 0 tokens; the LLM variant that scored best costs 564 ms and 171 tokens.
  • Guidance helps some judges and not others. It lifts the LLM-number judge from 0.358 (worse than a coin flip, a prior that is wrong in the other direction from Part 5’s) to 0.697, and Laya from 0.430 to 0.604. It does nothing for the token-probability LLM (0.671 bare, 0.643 guided, inside each other’s intervals). In the first version of this post guidance helped every judge. Here the model’s own prior about brightness is closer to right, so the bare token-probability judge is already decent.
  • The fine-tuned and multilingual checkpoints do not transfer. typed-decisions was trained on other typed decisions (invoices, security incidents, customer service) and sits at chance (0.478). multilingual is below chance (0.378). They flag every box at the dev threshold (513 of 513).
  • Laya’s probabilities are honest and nearly useless. Every guided Laya probability on the held-out boxes sits between 4.7% and 49.6%, the median for real boxes is 13.9% and for false alarms 12.5%. At 50% it flags nothing at all, so its accuracy of 84% is just the majority class. At the threshold I picked on dev it flags 223 boxes and 46 are real (precision 20.6%, recall 56.8%).
  • The best calibrated judge is the LLM writing a number, with a calibration error of 0.013 and a Brier skill of +0.21 (how much better than always predicting the base rate), the only judge that is clearly better than that baseline. In the first version the LLM’s numbers were overconfident. That changed, and I do not know why: same model, same prompt style, different data. Fitting one temperature on dev to recalibrate does not help it (+0.17).
  • It does offer a different operating point. At the dev threshold (0.30) the LLM number flags 64 boxes and 37 are real: precision 57.8%, recall 45.7%. The rule cannot offer that, because it only says yes or no, and its one point is 290 flagged, 65 real (precision 22.4%, recall 80.2%). Whether you want the first or the second is a decision about how many false alarms a human can tolerate, not a ranking-score question.

Here are all 1,059 boxes and all seven judge variants. Pick one, drag the threshold, and watch the dev and held-out rows move apart:

0%100%probability the judge gave (held-out boxes, green = real defect)
boxes flagged real among flagged of real precision recall
Every one of the 1,059 boxes is here with its real label. Dev images are where thresholds are chosen, held-out images are the test. Only 81 real boxes are in the held-out set, so one box moves precision by a point or more.

0.5 Experiment B: reading the notes

The second call is the opposite kind: free text in, typed facts out. Part 7’s report node writes a note per flagged image. I took all 520 notes the model wrote in Part 7’s six model runs, and asked three questions about each note, with the answer taken from the structured record the note was written from, not from labelling the prose: was the image escalated? is any kept box uncertain? how many boxes were kept?

Judge (520 real notes) Escalated? Uncertain? How many kept? One call
Chance (majority class) 0.554 0.562 0.558
Typed model, Laya 0.990 0.871 0.681 21 ms (31 ms for the count)
Typed model, typed-decisions 0.973 0.925 0.623 21 ms
Typed model, multilingual 0.933 0.627 0.585 9 ms
LLM, writes a number 0.715 0.933 0.937 509 to 2,711 ms
LLM, token probability 0.771 0.931 0.925 157 to 225 ms

The pattern is different from the first run, and the difference is the most interesting part. On escalated? the small typed model is nearly perfect (99%) and the language model is poor (71.5% and 77.1%). On counting (“how many boxes does this note say were kept?”) the typed model is clearly worse (68% against 92.5 to 93.7%). The LLM’s escalate errors are almost all one mistake: of its 148 wrong answers (writing a number), 146 were “yes, escalated” on a note whose record says it was not. Many of those notes say, in so many words, “received a review verdict”, and it still gave 0.8 or 0.9. I did not test whether a different wording of the question fixes that. So the boundary is not “typed models wobble at counting” alone: a language model can fail a tick-box question too, and a typed model can ace it, depending on how the question sits next to the text.

0.6 Putting it to work: a guard on the report note

Part 7 found one problem in the model-written notes that the id check cannot see: wording. A natural use for a fast, free typed reader is a guard: after the model writes the note, let Laya read it, and if what it reads disagrees with the record, throw the note away and use the template.

I ran that on all 520 notes with Laya’s answers to the two yes-or-no questions. It agreed with the record on 448 notes and disagreed on 72 (13.8%). Then I audited the 72. I read the first 36 in full next to their records, and the other 36 by their opening lines plus a keyword scan for “uncertain”, “low confidence” and “escalat”. The categories are assigned by rules that check the note text, and video_facts_p8.py asserts them:

  • 65 were Laya calling a perfectly good note uncertain, where the note says “confident defect” and names a box at confidence 0.6 to 0.9. The record has no uncertain box, and the note never says “uncertain”. Laya was wrong.
  • 5 were Laya reading “escalated” into a note that says “received a review verdict” in so many words. Laya was wrong.
  • 2 are ambiguous: the note says the image was escalated because too many boxes survived, which is not the same as an uncertain box, and Laya said an uncertain box exists.
  • 0 were real flaws in a note, meaning a case where the record says uncertain or escalated and Laya said no.

So 70 of the 72 flags were the guard’s own mistakes, 2 were arguable, and the guard found no real problem. In the first version it found two real omissions; here the notes did not contain one that Laya caught. The existing id check from Part 7 flagged none of the 520 notes. The reason is a base rate. When the writer is almost always right, a checker that is 86% right on these questions mostly flags good notes. I did not audit the 448 notes it passed, so I cannot tell you how many problems it missed. As built, I would not ship this guard. A tuned or fine-tuned checker, or a stricter threshold on its probabilities, might change that. I did not test either.

0.7 Jev: what I could not do

The roadmap asked for Jev against Laya. Jev is TypeSafe’s hosted “System One” model, in early access. Its public page says it returns typed decisions with calibrated probabilities and lists input at $42 per billion tokens (that is $0.042 per million, the same figure Laya’s README quotes). The pages I could read do not document the HTTP API or how to get a key, and I did not have one. I did not sign up for early access on anyone’s behalf.

What I did instead: typed_models.py has a JevJudge that posts to /v1/systemone with a state and questions, the request shape Laya’s own laya-serve says it mirrors. That is the unverified part: I wrote it from Laya’s README and Jev’s public page, and it has never been run. If you have access, set TYPESAFE_BASE_URL and TYPESAFE_API_KEY and run uv run python typed_eval.py jev AB; the two experiments would then produce a Jev row in exactly the same tables. Until someone does, any claim about Jev in this series is a claim about its page, not about its behaviour. That includes the third-party numbers in Laya’s README, which I did not reproduce.

0.8 So which tool for which step?

If the step is… Use Because (measured here)
numbers you can write a rule for a rule indistinguishable from the best model on the box call (0.641 against 0.697, interval includes zero), 40 ns, 0 tokens, exact and testable
reading text into a typed answer a small typed model, checked on your own question 21 ms and no tokens; 99% on “escalated?” where the LLM got 72 to 77%; worse at counting (68% against 93%); zero-shot it needs the guidance in the question
writing the language model, with code owning the facts the only one that can write; 2.4 s and 243 tokens per note

And in all cases, measure on your labeled data with held-out cases, and do not stop at calibration. A model that never says “yes” can have a lovely calibration error.

In the pipeline from Part 7, this says: keep measure, rules and decide as code; use a rule for the box call (the best LLM judge was not resolvably better, and costs 216 to 564 ms and 146 to 171 tokens per box; the typed models did not match it); keep the LLM for report. A typed model reading text is a good fit for tick-box questions about free text, and in my tests it did not earn a place guarding the note. One thing the LLM-number judge does offer is a well-calibrated probability, which is what a review-versus-escalate threshold (Part 9) would want.

0.9 Jargon, explained simply

Typed model. A model that answers a fixed kind of question (yes/no, pick one, score) with a probability, and cannot generate free text.

Daily-life version: a multiple-choice exam versus an essay. You can mark one by machine.

Calibration. Whether a stated probability matches reality: of all the things said to be 80% likely, about 80% should happen.

Daily-life version: a weather forecaster who says “30% rain” on 100 days, and it rains on about 30.

ECE (expected calibration error). The average gap between stated confidence and actual accuracy. Lower is better, and a model that always says “about 6%” on a 6%-prevalence problem scores well.

AUC (ranking score). Of all pairs of one real box and one false alarm, how often the real one gets the higher probability. 0.5 is a coin flip, 1 is perfect.

Daily-life version: shuffle a pile of sick and healthy patients’ charts: does the test rank the sick ones higher?

Brier skill. How much better your probabilities are than always predicting the base rate. Zero means no better; negative means worse.

Token probability (logprob). The model’s own probability for the next word. Reading it is different from asking the model to say a number.

Base rate. How common the thing is. A checker that is 92% right is poor at finding something that happens 1% of the time.

Daily-life version: an airport scanner that is right 92% of the time, in a queue where nearly nobody has a weapon.

Guard. A cheap check that reads another step’s output before it is trusted.

0.10 What I did not do

  • Jev was not run. No API key. The adapter exists and is unverified. Nothing here says how Jev compares.
  • Laya was zero-shot. I used the three shipped checkpoints as they come and did not fine-tune, which Laya’s own README says is where accuracy comes from. A fine-tuned Laya might beat these numbers, or might not.
  • 81 real held-out boxes. Every precision and ranking figure moves by a point or more if a box moves. I bootstrapped the ranking scores over images and every difference from the rule has an interval that includes zero. No interval on the other columns.
  • One dataset, and no inspector notes. The dataset has no human-written notes, so experiment B reads the pipeline’s own model-written notes. The roadmap’s example, reading an inspector’s note for a solder void, I did not test. That is the setting where I would most expect a typed model to earn its place.
  • The answers in B come from the record. If the model wrote a note unfaithfully, “the note says X” and “the record says X” could differ and I would count it against the reader. That is why I audited the 72 disagreements, but I read 36 in full and the other 36 by their opening lines and a keyword scan, and I did not read the 448 agreements.
  • The LLM’s weak “escalated?” score is untested for cause. I saw that 146 of its 148 errors are false “yes”, and I did not try another wording of the question.
  • The thresholds were chosen on 546 dev boxes with 122 real ones, then applied to the held-out boxes.
  • Latencies are warm, sequential, on a GPU shared with two vLLM servers; the CPU number depends on its thread count (20 here). The LLM numbers include a network hop to a local server.
  • I used one local LLM (Qwen3-Coder-30B) at temperature 0 for the judge calls, with one prompt per question. A different prompt or model could move its numbers, as the token-probability bug showed.
  • The videos are made with two tools on purpose: Manim for the data chapter (real per-box probabilities), Motion Canvas for the card-and-flow chapters, all in my cloned warm voice. They were checked by transcript and extracted frames, not by ear.

0.11 Reproduce this yourself

Show reproduce commands 
cd quarto_blog_hasan/notebooks/agentic-zero-to-advanced
uv sync                                          # laya (and torch) are in pyproject.toml now
# local vLLM on :8000 serving qwen3-coder, as in Parts 3-7
uv run python typed_eval_data.py                 # the two evaluation sets (needs Part 7's pipeline_graph.py run first)
uv run python typed_eval.py band A               # the rule
uv run python typed_eval.py laya-english,laya-typed-decisions,laya-multilingual AB   # GPU, about 5 min each
uv run python typed_eval.py llm-verbalized,llm-logprob AB                            # ~10 min each
uv run python typed_eval.py jev AB               # only with TYPESAFE_BASE_URL and TYPESAFE_API_KEY set
uv run python typed_costs.py                     # what each kind of tool costs per call
uv run python typed_summary.py                   # the tables above
uv run python typed_bootstrap.py                 # the intervals on the box-call ranking scores
uv run python typed_guard.py laya-english        # the 72 disagreements, printed next to their records

0.12 What’s next

Further reading: Laya’s README documents its question types and fine-tuning notebook, and Jev’s page describes the hosted alternative.

Part 9 makes escalate real: the graph pauses before the risky step and waits for a human, using the checkpointer from Part 6. The probabilities from this post are what a threshold there would be set on, which is exactly why it matters that Laya’s were honest and flat, and the LLM’s were sharp and overconfident.

Reuse

Citation

BibTeX citation:
@online{goni2026,
  author = {Goni, Hasan},
  title = {Agentic {Developer:} {Zero} to {Advanced,} {Part} 8: {Tool}
    {Design}},
  date = {2026-10-17},
  langid = {en}
}
For attribution, please cite this work as:
Goni, Hasan. 2026. “Agentic Developer: Zero to Advanced, Part 8: Tool Design.” October 17.
Free email course · 7 days

Industrial Vision Foundations

AI can write the code, but someone still has to know which method fits a real inspection line. One short lesson a day for 7 days on the classical computer vision techniques that still run production systems.

Runnable code in every lesson. No spam, unsubscribe any time.

Questions, corrections or feedback?

Found a mistake, or want a topic covered next? Leave a comment below (sign in with GitHub), or write to hasanme1412@gmail.com. I read everything.