This post is part of the Agentic Developer: Zero to Advanced series. It follows on from Part 2 — the dataset and the classical baseline.
Part 2 ended with a detector that works, badly. Precision 0.165, recall 0.244 on the held-out images, honest numbers, no LLM anywhere. This post adds the first agentic ingredient: tool calling. The classical detector becomes a Python function the model can invoke. You send one request; the model calls the tool, reads the structured JSON it returns, and writes a short explanation a human operator could actually use.
Still no planning loop, no memory, no LangGraph. Just: ask, tool runs, model explains.
Rewritten on 2026-10-18 on the synthetic void dataset (see Part 2), because the first version used GDXray images that I may not redistribute. The tool also changed: the old one handed the model the ground truth. A deployed tool has no answer key, so this one does not, and the model can no longer say “false positive” with confidence. That is the honest setting.
0.1 Why wrap the detector at all
A plain LLM asked “are there voids in this X-ray?” has to guess. It has never seen the image bytes in Part 3’s setup, only your words. Tool calling fixes that gap: the model decides when to run the real detector, gets back real coordinates and counts, and only then writes its verdict.
Think of the difference between asking a colleague “does this look defective?” versus handing them the image and a measurement report from the actual inspection algorithm. The second conversation starts from evidence.
0.2 The tool: structured detector output
The tool is a normal Python function with a docstring. aisuite reads that docstring and builds the JSON schema the model uses to call it, the same pattern as the Agentic Development tool-use post, scoped to the baseline from Part 2.
Show inspect_xray()
def inspect_xray(image_id: str) -> str:
"""Run the classical top-hat void detector from Part 2 on one X-ray image.
Args:
image_id: Image id such as test_00009 (split name, underscore, five digits).
Returns:
JSON string with the candidate boxes (pixels), each with its size and strength, where
strength is the peak response divided by the image's noise level.
"""
split, i = parse_image_id(image_id)
gray = vd.load_image(split, i)
boxes, st = detect_candidates(gray, K_DEV, return_stages=True)
sigma = max(st["sigma"], 1e-3)
cands = [{"x1": x1, "y1": y1, "x2": x2, "y2": y2, "width": x2 - x1, "height": y2 - y1,
"strength": round(float(st["response"][y1:y2, x1:x2].max()) / sigma, 1)}
for x1, y1, x2, y2 in boxes]
return json.dumps({"image": ..., "part_area_px": ..., "candidate_boxes": len(cands),
"candidates": cands, "threshold": f"{K_DEV} x noise level", ...}, indent=2)On test_00009, the held-out image from Part 2, the tool returns this (real output from detector_tool.py, abbreviated to one line per box):
{
"image": "test_00009",
"image_size": {"width": 224, "height": 224},
"part_area_px": 26297,
"candidate_boxes": 3,
"candidates": [
{"x1": 99, "y1": 0, "x2": 122, "y2": 5, "width": 23, "height": 5, "strength": 11.2},
{"x1": 78, "y1": 219, "x2": 88, "y2": 224, "width": 10, "height": 5, "strength": 10.3},
{"x1": 135, "y1": 222, "x2": 138, "y2": 224, "width": 3, "height": 2, "strength": 8.8}
],
"threshold": "8.0 x noise level",
"detector": "classical top-hat baseline (Part 2)"
}Three things are worth noticing. There is no ground truth in it. Every candidate sits at the image border (y = 0 or y = 224), where the part is cut off by the frame. And in Part 2 this image had three real voids that the detector missed entirely, so all three candidates are false alarms. Neither the tool nor the model knows that. Part 7 adds a border rule for exactly this.
0.3 One LLM call, three internal steps
Under the hood, aisuite with max_turns=3 does a short loop:
- You send the user message.
- The model returns a tool call (
inspect_xray). - Python runs the detector, feeds the JSON back.
- The model writes the final explanation.
You write one create() call; the client handles the hand-off.
0.4 Where the API key usually lives
Most tutorials start with a .env file next to your script:
first_tool_call.py loads that file on startup (without overwriting variables you already exported in the shell). aisuite passes OPENAI_API_KEY straight through to the OpenAI client — that is the normal cloud path.
I’m not using that here. I have Qwen3-Coder-30B running locally through vLLM, which exposes an OpenAI-compatible HTTP API on port 8000. Same aisuite code, different endpoint:
Copy .env.example in that folder if you want a starting template. The openai: prefix is aisuite’s provider tag; everything after the colon must match the model name your vLLM server advertises.
Show first_tool_call.py
import aisuite as ai
from detector_tool import inspect_xray
client = ai.Client()
response = client.chat.completions.create(
model="openai:qwen3-coder", # or openai:gpt-4o-mini for cloud
messages=[
{"role": "system", "content": (
"You are an industrial X-ray inspection assistant. The tool reports candidate voids; "
"it has no answer key, so do not claim a candidate is a real defect. "
"Always call inspect_xray before answering."
)},
{"role": "user", "content": (
"Inspect this frame and tell me whether it looks like a pass or needs human review. "
"Image id: test_00009. Keep it under 100 words."
)},
],
tools=[inspect_xray],
max_turns=3,
)
print(response.choices[0].message.content)0.5 What the local model said
I ran this against my local vLLM server (qwen3-coder, the id returned by curl http://localhost:8000/v1/models). Given the tool result above, here is the explanation for test_00009:
The detector found three candidate voids in test_00009. Two are near the top and one near the bottom. All exceed the detection threshold of 8.0 times noise level. Recommend human review to confirm if these are actual defects or false positives. The part area is 26,297 pixels.
The count is right and the recommendation is sensible, and there is already a mistake in it: the tool’s boxes put one candidate at the top (y = 0) and two at the bottom (y = 219 and y = 222), not two and one. The model reasoned about positions it could have read exactly and got them wrong. That is why I checked more than one image.
0.6 Checking it on ten held-out images
I ran the same call on the first ten held-out images in order (test_00000 to test_00009, not picked by outcome) and checked each answer two ways: automatically, does the explanation quote the number of candidates the tool returned, and by reading, does it claim anything the tool did not say? The raw answers are in out/consistency.json.
| Check | Result |
|---|---|
| Quotes the correct candidate count | 10 of 10 |
Calls the candidates “dark” (3 of 10: test_00000, test_00006, test_00009) |
wrong, a void is brighter, and the tool never mentions polarity |
Says strengths are “below the threshold” (test_00007) |
wrong, every returned candidate is above it by construction |
Says “the image seems to pass” with 2 candidates raised (test_00007) |
contradicts its own recommendation to review |
Gets the position of the candidates wrong (test_00009) |
wrong, see above |
So the part the tool supplies (the count) comes through reliably, and the part the model adds (what the candidates look like, whether they pass) is where it makes things up: 4 of the 10 batch explanations (test_00000, test_00006, test_00007, test_00009) contain at least one claim I can show is false or inconsistent. The position error on test_00009 came from the single call above, whose text differs from the batch run’s, because sampling is not deterministic. Ten images is a small sample, and I wrote the checks after reading the answers, so treat the rates as an illustration of the kind of error, not a measurement.
That is the payoff and the warning in one place. The model did not invent defect counts. It read them from the tool. But as soon as it is allowed to describe what it did not measure, it fills the gap with plausible words. Part 4 shrinks that freedom by forcing the answer into fields, and Part 7 takes the pass/review decision away from the model altogether.
0.7 What this is not (yet)
- Not an agent. There is no replanning, no second tool choice, no “try a different threshold.”
- Not structured output. The final answer is free-form prose. Part 4 tightens that to reliable JSON fields (defect type, severity, confidence).
- Not better detection. The underlying detector is still Part 2’s classical baseline. The LLM layer adds interpretation, not accuracy.
- Not a measured error rate. The ten-image check above shows the kinds of mistakes, with a sample too small and a checklist too informal to quote as a rate.
0.8 Jargon, explained simply
Tool calling. The mechanism by which an LLM invokes a real function and receives a real result back, instead of guessing from memory. Think of asking someone to check the thermometer instead of estimating the temperature from how the room feels.
Schema. A machine-readable description of what a function expects and returns, built from the Python docstring so the model knows how to call the tool correctly.
max_turns. A safety cap on how many back-and-forth steps (tool call → result → response) the client allows in one request. Suppose you set it to 3: the model gets at most one tool call and one final answer in this post’s setup.
Structured output (next post). A response forced into a fixed JSON shape so downstream code can parse it reliably — not the free-form paragraph above.
vLLM. A local inference server that serves an LLM behind an OpenAI-compatible REST API. Think of it as running your own private api.openai.com on localhost:8000 — same request format, no data leaves the machine.
0.9 Reproduce this yourself
Show reproduce commands
Cloud .env:
Local vLLM .env (what I used for the captured output above):
The script always writes out/tool_result.json. With an API key set (real or dummy for vLLM), it also writes out/llm_explanation.txt with the live model response.
The dataset must be built first (see the commands in Part 2). For local inference, start vLLM first and confirm the model id with curl http://localhost:8000/v1/models. The script also runs the ten-image check, which makes eleven model calls in total.
0.10 What’s next
Part 4 keeps the same tool but forces the LLM’s final answer into reliable JSON — defect type, severity, confidence — so the explanation is machine-parseable, not just readable.
Reuse
Citation
@online{goni2026,
author = {Goni, Hasan},
title = {Agentic {Developer:} {Zero} to {Advanced,} {Part} 3: {Your}
{First} {Tool-Calling} {LLM}},
date = {2026-09-14},
langid = {en}
}