This post is part of the Agentic Developer: Zero to Advanced series. It follows on from Part 4 – prompting and structured output. It opens Arc 2, the agent loop.
Rewritten on 2026-10-18 on the synthetic void dataset, with fresh runs. The first version used GDXray images that I may not redistribute, and it found that the agent did worse than the plain detector, because on that data the model’s prior about contrast was backwards. On this data the picture is different, and I report the different picture.
Arc 1 is done. We have a detector, a tool wrapper, and a model that returns clean JSON. Still no agent. Every call so far was one shot: input in, answer out.
This post builds the agent. No framework. It is a loop of about 40 lines. Then I run it on 200 dev and 200 held-out images and score it. Spoiler: it is more precise than the plain detector and finds fewer real voids, and that trade is the useful part, because then we get to look at where the agent’s decisions come from and what fixes help, with real numbers.
0.1 What “agent” means
People use the word for anything with an LLM in it. I use it for one thing: the model decides what happens next, and code does it.
Four pieces make that work:
- Planning. The model picks the next step. You don’t hardcode “call tool A, then tool B”.
- Tool use. The model asks for a function call. Your code runs it and hands back the result.
- Memory. The result goes back into the conversation, so the next decision can use it.
- Control flow. Something decides when to stop. Usually a loop with a step limit.
Daily-life version: a plumber at your door. You don’t give him a script. He looks, tries something, sees what happened, tries the next thing, and tells you when he is done. A plain LLM call is a plumber who answers through the letterbox without looking at the pipe.
0.2 The tools
Same detector as Part 2. Two changes.
First, I split it into small tools. find_candidates runs the detector. measure_candidate looks closer at one box: contrast against its surroundings, area, aspect ratio, distance to the image edge.
Second, no ground truth in any tool result. Part 3’s tool already left the answer key out, and this stays true here: no tool result carries a hit count or a label. An agent that sees the answer key is not being tested. The scoring code sees ground truth, the agent does not.
Show inspection_tools.py (the two tools)
notebooks/agentic-zero-to-advanced/inspection_tools.py
def find_candidates(image: str) -> dict:
"""Run the classical detector on one X-ray image. Returns candidate boxes with an id each.
The detector is noisy: most boxes are not real voids."""
image = norm(image)
gray = load_gray(image)
boxes = detect_candidates(gray, K_DEV)
_CANDIDATES[image] = boxes
return {
"image": image,
"image_size": [int(gray.shape[1]), int(gray.shape[0])],
"candidates": [
{"id": i, "box": [int(a) for a in b], "w": int(b[2] - b[0]), "h": int(b[3] - b[1])}
for i, b in enumerate(boxes)
],
}
def measure_candidate(image: str, candidate_id: int) -> dict:
"""Measure one candidate box: contrast against its surrounding ring (positive = brighter than
surroundings, negative = darker), area, aspect ratio, and distance to the nearest image edge."""
image = norm(image)
gray = load_gray(image)
boxes = _CANDIDATES.get(image)
if boxes is None:
find_candidates(image)
boxes = _CANDIDATES[image]
if not 0 <= candidate_id < len(boxes):
return {"error": f"candidate_id {candidate_id} out of range (0..{len(boxes) - 1})"}
x1, y1, x2, y2 = boxes[candidate_id]
h, w = gray.shape
return {
"candidate_id": candidate_id,
"contrast": round(contrast_delta(gray, x1, y1, x2, y2), 2),
"area_px": int((x2 - x1) * (y2 - y1)),
"aspect_ratio": round((x2 - x1) / max(1, (y2 - y1)), 2),
"edge_distance_px": int(min(x1, y1, w - x2, h - y2)),
}The docstring is not decoration. It is what the model reads to decide when to call the tool. A bad docstring is a bad tool, same as a bad function name in a codebase.
0.3 The loop
This is the whole agent. A for loop, a list called messages, and one extra tool, submit_report, that means “I’m done”.
notebooks/agentic-zero-to-advanced/agent_loop.py
def run_agent(image: str) -> dict:
messages = [{"role": "system", "content": SYSTEM},
{"role": "user", "content": f"Inspect image {image}."}]
trace, kept_ids, report, t0, tokens = [], [], None, time.time(), 0
for step in range(1, MAX_STEPS + 1):
resp = client.chat.completions.create(
model=MODEL, messages=messages, tools=TOOL_SPECS + [SUBMIT_SPEC], temperature=0.2)
tokens += resp.usage.total_tokens
msg = resp.choices[0].message
messages.append(msg.model_dump(exclude_none=True))
if not msg.tool_calls: # model talked instead of acting: nudge it once per step
trace.append({"step": step, "says": msg.content, "calls": []})
messages.append({"role": "user", "content": "Use a tool, or call submit_report if you are done."})
continue
step_calls = []
for call in msg.tool_calls:
args = json.loads(call.function.arguments or "{}")
if call.function.name == "submit_report":
report, result = args, {"ok": True}
elif call.function.name in TOOL_FUNCS:
try:
result = TOOL_FUNCS[call.function.name](**args)
except Exception as exc: # tool errors go back to the model, they don't crash the loop
result = {"error": str(exc)}
else:
result = {"error": f"unknown tool {call.function.name}"}
step_calls.append({"tool": call.function.name, "args": args, "result": result})
messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})
trace.append({"step": step, "says": msg.content, "calls": step_calls})
if report:
break
... # scoring against ground truth, see the repoTwo details matter more than they look.
messages.append(...)is the memory. There is no other state. If it is not in that list, the model does not know it.- Tool errors go back to the model, they don’t crash the loop. If the model asks for
candidate_id=99, it gets{"error": ...}and can try again. Same idea as Part 4’s retry, now inside the loop.
0.4 One real run, step by step
Image test_00020, the first held-out image with exactly four candidate boxes (picked by that rule, not by outcome). Click through what the agent did. Everything below is the real trace from the run, not a mock-up.
Things to notice:
- The first step is
find_candidates. Nobody told it to in code. The system prompt said so, and it followed. - It measures one box per step. It could have asked for all four in one go. Four steps instead of one. That costs time. More on this in Part 8.
- It measured candidates 0, 1 and 2 and never measured candidate 3, then submitted. Nothing in the loop objected. Hold that thought, it comes back as the verification gap.
- The last step is
submit_report. That is the stop signal. The loop ends because the model said so. - It kept 0 and 1 and dropped 2 (“too close to the edge and has low contrast”). Candidate 2 touches the right border (
edge_distance_pxis 0), and the model noticed. On this image that was right: one of the two it kept was a real void, the other a false alarm, and no real void was missed.
0.5 Scoring it
The protocol from Part 2: dev is the val split, held-out is the test split. The agent costs real model calls, so I ran the first 200 images of each (in order, not picked by outcome): 200 dev images with 420 real voids and 200 held-out images with 387. Model: Qwen3-Coder-30B on local vLLM, same as Part 3. A kept box is a hit when it overlaps an exact void box with IoU of at least 0.2, the same rule as Part 2. The plain detector row keeps every candidate at the Part 2 threshold.
| Held-out (200 images, 387 voids) | Boxes kept | Real found (TP) | False alarms (FP) | Missed (FN) | Precision | Recall |
|---|---|---|---|---|---|---|
| Plain detector (Part 2) | 513 | 81 | 432 | 306 | 15.8% | 20.9% |
| Agent loop (this post) | 278 | 63 | 215 | 324 | 22.7% | 16.3% |
On dev the same two rows read 22.3% / 29.0% for the detector and 34.6% / 23.3% for the agent, the same shape at a different level: the agent keeps about half the boxes, gives up roughly a fifth of the real voids and gets more precise. The cost per image:
| Per image, average (held-out) | Value |
|---|---|
| Steps | 3.8 |
| Tool calls | 3.85 |
| Tokens | 3,862 |
| Wall time | 27.0 s |
The wall time is high because the model server is shared with other work and eight runs went in parallel, so treat it as an upper bound. All 400 runs finished with submit_report. None hit the step limit.
0.6 Where the agent’s decisions come from
An agent loop adds decisions, not information, so the question is whether the measurements it sees carry any. Before writing any guidance I measured that on the dev split only, all 600 images: 1,736 candidates, 321 of them real. Mann-Whitney AUC per measurement (0.5 is a coin flip):
| Measurement | AUC | Median, real void | Median, false alarm |
|---|---|---|---|
| contrast (signed) | 0.74 | 19.6 | 6.8 |
| area (px) | 0.68 | 100 | 42 |
| distance to edge (px) | 0.68 | 49 | 26 |
| absolute contrast | 0.67 | 23.2 | 13.9 |
| aspect ratio | 0.46 | 1.0 | 1.0 |
Bands show it more plainly. Share of candidates that are real, by band, on dev:
| Contrast | Real | Distance to edge | Real | Area | Real | ||
|---|---|---|---|---|---|---|---|
| below 0 (n=544) | 10% | 0 px (n=525) | 0.2% | under 20 (n=415) | 4% | ||
| 10 to 15 (n=245) | 12% | 1 to 5 (n=55) | 7% | 40 to 80 (n=311) | 22% | ||
| 20 to 30 (n=196) | 36% | 10 to 20 (n=96) | 26% | over 300 (n=187) | 31% | ||
| 30 to 60 (n=121) | 71% | over 20 (n=1,028) | 27% |
So brighter really does mean more likely a real void, the model’s common sense is right this time, and 525 of the 1,736 candidates (30%) touch the image border and exactly one of them is real. That is why the agent gets more precise by simply reading measure_candidate: the information was in the tool output. In the first version of this post, on GDXray, contrast was backwards (AUC 0.41) and the same common sense made the agent worse. The lesson is the same, only the sign changed:
An agent loop does not add information. It only adds decisions.
If the evidence is strong, the loop makes good decisions from it. If the evidence is weak or misleading, it makes confident decisions on the same weak evidence and charges you tokens for it. And there is a ceiling the loop cannot lift: it can only choose among the boxes the detector raised. The plain detector found 81 of 387 real voids, so no filtering of its output can find more than 21%. The agent’s 16.3% is a filter on a 20.9% ceiling. A better detector, or a model that looks at the actual image (Part 17), is the only way up.
0.7 Why agents fail: three buckets
Cemri et al. read over 1,600 traces from seven multi-agent LLM systems and sorted the failures into three buckets (paper). Their systems are multi-agent and mostly write software, ours is one agent looking at X-rays, so I use the buckets, not their numbers.
- Design issues. The system was set up badly. Vague role, no stopping rule, missing context.
- Misalignment. The parts don’t share what they know. Wrong assumptions made instead of asking.
- Verification gaps. Nothing catches the error. The check is superficial or missing.
Daily-life version: a restaurant order goes wrong. Either the order slip was unclear (design), the waiter and the kitchen assumed different things (misalignment), or nobody looked at the plate before it left the kitchen (verification).
Our agent from above shows the first and third, and the second only in a thin form:
- Design. The system prompt never said what a real void looks like or how this detector fails. The model filled the gap with its own prior. The prior was decent here (brighter is better), but it does not know that 30% of all candidates sit on the image border and are nearly never real, and it kept some of them.
- Verification. Nothing in the loop checked the answer against anything.
submit_reporttook whatever the model gave it, including a report that skipped a candidate (test_00020above). - Misalignment. The model assumed instead of asking, but it had nobody to ask. I did not see the classic version of this, agents contradicting each other, because there is only one agent. It shows up in Part 10 when there are several.
So two fixes to try. Both run on the held-out images only, and I wrote the guidance from the dev split (val), which the held-out split never touched. All numbers in this section are from the 200 held-out images, 387 real voids.
0.7.1 Fix 1: design, a better role spec
The dev bands above are the guidance. Border boxes: 1 real in 525. Contrast above 30: 7 in 10. Area under 20 px: 1 in 25. I wrote exactly that into the prompt, plus a rule to measure everything before keeping anything.
notebooks/agentic-zero-to-advanced/agent_fixes.py
SYSTEM_DESIGN = SYSTEM + """
What the labeled dev data says about this detector:
- A box touching the image border (edge_distance_px = 0) is almost never real: 1 in 525.
- Contrast is the best signal. Above 30 about 7 in 10 boxes are real, 20 to 30 about 1 in 3,
below 15 about 1 in 10 or fewer.
- Boxes smaller than 20 pixels in area are almost never real: about 1 in 25.
- Measure every candidate before you keep or drop it. Never keep an id you did not measure.
- Finish with verdict "review" if you kept anything, "pass" if you kept nothing."""No code change in the loop. Only the prompt. That is the point of a design fix.
0.7.2 Fix 2: verification, check the report before accepting it
Part 4 validated the JSON shape. That is the “superficial check” the paper complains about: it confirms the output is well formed, not that it is right. Here the check compares the report to what actually happened in the loop. If something is off, the problems go back to the model as the tool result, and it tries again (twice at most).
notebooks/agentic-zero-to-advanced/agent_fixes.py
def verify_report(report: dict, measured: set[int], n_candidates: int) -> list[str]:
problems = []
kept = report.get("kept_ids", [])
bad = [i for i in kept if not 0 <= i < n_candidates]
if bad:
problems.append(f"kept_ids {bad} are not valid candidate ids (valid: 0..{n_candidates - 1})")
unmeasured = [i for i in kept if i not in measured and 0 <= i < n_candidates]
if unmeasured:
problems.append(f"you kept ids {unmeasured} without measuring them. Measure them first.")
unchecked = [i for i in range(n_candidates) if i not in measured]
if unchecked:
problems.append(f"candidates {unchecked} were never measured. Measure them or say why not.")
if report.get("verdict") == "pass" and kept:
problems.append("verdict is 'pass' but kept_ids is not empty")
if report.get("verdict") == "review" and not kept:
problems.append("verdict is 'review' but kept_ids is empty")
return problems
# inside the loop, when the model calls submit_report:
problems = verify_report(args, measured, n_cand) if verify else []
if problems and rejections < MAX_REJECTIONS:
rejections += 1
result = {"ok": False, "problems": problems, "fix": "Fix these and call submit_report again."}
else:
report, result, done = args, {"ok": True}, TrueI ran this check over the 400 reports from the original agent, without changing anything. It would have bounced 70 of 400 (36 dev, 34 held-out). All 70 for the same reason: candidates the agent never measured. No invented ids, no pass/review mismatch, and it never kept a box it had not measured. So the agent was skipping work on 17% of images, and nothing noticed.
0.7.3 Results
| Held-out (200 images, 387 voids) | Boxes kept | Real found (TP) | False alarms (FP) | Precision | Recall | Avg tokens / image |
|---|---|---|---|---|---|---|
| Plain detector | 513 | 81 | 432 | 15.8% | 20.9% | – |
| Original agent | 278 | 63 | 215 | 22.7% | 16.3% | 3,862 |
| + design fix | 211 | 66 | 145 | 31.3% | 17.1% | 4,901 |
| + verify fix | 343 | 68 | 275 | 19.8% | 17.6% | 5,348 |
| + both | 225 | 67 | 158 | 29.8% | 17.3% | 6,120 |
Three runs (verify and both each ended 3 images without a report: the step limit, 197 of 200 finished).
What I read from this:
- The design fix did the real work. Precision 22.7% to 31.3% with the same number of real voids found (63 to 66). It dropped 70 false alarms and lost nothing. With counts this size that is about two standard errors, so it is believable but I would not call it settled.
- The verify fix alone did not help precision. It made the agent measure more (about 0.17 rejections per image) and the tokens rose by a third, but it kept the same kind of boxes, with better paperwork. A check that tests process cannot fix a missing piece of knowledge.
- Both together were not better than design alone. 29.8% against 31.3% precision, 67 against 66 real voids. That is noise. Do not read it as “verification hurts”.
- Nothing gets above the plain detector’s recall. Every agent sits at 63 to 68 real voids out of the detector’s 81. The agent cannot find what the detector never boxed, and it loses a few on the way.
Caveats, because they matter: one run each, temperature 0.2, and a few dozen true positives per row. The direction is believable. The exact numbers are not stable. And the guidance was written for this detector on this data. Move to a different detector and it is wrong again.
0.7.4 The control: the same guidance as a plain rule
The design fix is a bundle of numbers read off the dev data. So the obvious control is to apply those numbers as code, with no model. rule_control.py keeps a box if it does not touch the border, has area at least 20 and contrast at least c. At c = 20 on the held-out images:
| Held-out (200 images) | Boxes kept | Real found | Precision | Recall |
|---|---|---|---|---|
| Rule, contrast >= 20 | 77 | 45 | 58.4% | 11.6% |
| Rule, contrast >= 5 | 171 | 65 | 38.0% | 16.8% |
| + design fix (agent) | 211 | 66 | 31.3% | 17.1% |
A rule is a point on a curve, and I swept c to find the one at the agent’s recall. At that recall (16.8% vs 17.1%) the rule’s precision is 38.0% against the agent’s 31.3%. In fairness: I chose that threshold by looking at held-out numbers, which the agent did not get to do, and the gap is one configuration. But it says the agent is not buying anything here that three thresholds do not. That is a good question to carry into Part 8.
0.7.5 The check that would have caught it: a gate on labeled data
The verify_report check passes agents that are wrong, as shown. The check the paper asks for is at task level: does the whole thing do the job? Here that means: before trusting the agent, score it on labeled images and compare it to the thing it replaces.
notebooks/agentic-zero-to-advanced/fixes_analysis.py
The 0.8 is my choice, not a standard, and I set it in the first version of this post, before these runs. Result on the held-out images: the original agent fails (recall 16.3% is 78% of the detector’s 20.9%, under the 80% line), and all three fixed agents pass (82%, 84% and 83%).
Look at those margins: they are one or two points either side of the line, on one run each. A different seed could flip the verdict, and I would not trust this gate to separate “good” from “a little worse” at this sample size. What the gate does catch is the property it was built for: every agent here finished without errors and returned valid output, so a superficial check would have shipped all four, and the gate at least makes you look at precision and recall against the thing you are replacing.
Untested idea for the misalignment bucket: give the agent a defect_stats tool that returns the dev-set numbers, so it can look up the data instead of guessing. I did not run it.
0.8 What the loop does buy you
Not accuracy, here. Three other things.
- A trace. Every decision is a line you can read. The viewer above is just that list.
- Error recovery. A bad tool call becomes a message, not a crash.
- A place to put control. Step limits, approval gates, cost caps. All of Arc 2 and 5 hang off this one
forloop.
0.9 Jargon, explained simply
Agent. A program where a model decides the next action and code carries it out, repeatedly, until the model says it is done.
Daily-life version: the plumber again. A chatbot tells you how plumbers work. An agent comes over and fixes the pipe.
Tool call. The model returns a function name and arguments instead of text. Your code runs the function.
Daily-life version: a chef calling out “knife!” to the kitchen hand. The chef decides. The hand does it.
Agent loop. Ask the model, run its tool, add the result to the history, ask again.
Daily-life version: playing 20 questions. Each answer changes what you ask next, and you stop when you can name the thing.
Context / messages list. Everything the model can see right now. The only memory the agent has.
Daily-life version: a notepad you reread in full before every step. If you did not write it down, it is gone.
Failure buckets (MAST). Design issues, misalignment, verification gaps. A way to sort why an agent failed so you know what kind of fix to reach for.
Daily-life version: the three ways a restaurant order goes wrong: unclear slip, waiter and kitchen assuming different things, or nobody looked at the plate.
Verifier / gate. Code that checks an agent’s output before you accept it. A gate tests the whole job on labeled data, not just whether the output is well formed.
Daily-life version: a taste test before the dish leaves the kitchen, not just checking the plate is clean.
Step limit. A hard cap on loop iterations so a confused agent cannot run forever.
Daily-life version: a kitchen timer. Not for the dish, for you.
AUC. How well a single number separates two groups. 1.0 is perfect, 0.5 is a coin flip, below 0.5 is backwards.
Daily-life version: guessing which of two people is taller by checking shoe size. Works if shoe size tracks height. Here it didn’t.
0.10 What I did not do
- No repeat runs. One run per variant at temperature 0.2. I have no estimate of run-to-run noise, and several comparisons above are inside it.
- No intervals. The percentages come from a few dozen to a few hundred boxes.
- Only 200 images per split. The agent costs model calls, so I used the first 200 of 600 images in each split. The plain detector and the rule were also scored on those 200 for a fair comparison.
- No second prompt wording, and no other model. The design guidance is one text.
- The rule’s threshold was swept on held-out numbers, which flatters the rule against the agent.
- The agents never saw an image, only measurements. A model that looks at the picture is Part 17.
0.11 Reproduce this yourself
Show reproduce commands
cd quarto_blog_hasan/notebooks/agentic-zero-to-advanced
uv sync
# build the dataset first (Part 2); local vLLM on :8000 serving qwen3-coder, same as Part 3. .env as in Part 4.
uv run python agent_loop.py # 200 dev + 200 held-out images, ~20 min, writes out/agent_loop_*.json
uv run python feature_check.py # the AUC table (dev only)
for v in design verify both; do uv run python agent_fixes.py $v; done # fix experiments, held-out images
uv run python rule_control.py # the no-model control
uv run python fixes_analysis.py # summary table and the gate
uv run python generate_arc2_diagrams.pyI ran it once. Numbers will move between runs (temperature 0.2, not 0). I would be surprised if the direction changed, but I did not test that.
0.12 What’s next
Further reading: Cemri et al., Why Do Multi-Agent LLM Systems Fail?. Section 4 has the three categories with examples, Section 5 has what helps.
Part 6 rebuilds this loop as a LangGraph graph: state, nodes, edges. Same idea, but the control flow stops being a for loop you wrote and becomes something you can draw, inspect, and pause.
Reuse
Citation
@online{goni2026,
author = {Goni, Hasan},
title = {Agentic {Developer:} {Zero} to {Advanced,} {Part} 5: {What}
{Makes} {It} an {Agent}},
date = {2026-10-03},
langid = {en}
}