Agentic Developer: Zero to Advanced, Part 4: Prompting and Structured Output

Tip

This post is part of the Agentic Developer: Zero to Advanced series. It follows on from Part 3 – your first tool-calling LLM.

Note

Rewritten on 2026-10-18 on the synthetic void dataset, with a fresh run on all 600 held-out images. The first version used GDXray images that I may not redistribute. The result is different in an interesting way: the structure is still perfect, and now I can show that the labels inside it are worthless.

In Part 3, we wrapped our classical computer vision detector as a tool and called it from an LLM. But the LLM response was still a free-form text explanation. This post introduces structured output – getting the LLM to return data in a predictable format (like JSON) that we can reliably parse and use in downstream code.

0.1 Why structured output matters

When an LLM gives you free-form text, you’re essentially getting a human-readable explanation. While helpful for humans, this makes it hard for code to consume the output automatically. In an agentic system, you want the LLM to return structured data that can be:

• Parsed programmatically
• Used as input for other tools
• Stored in databases
• Compared with previous outputs
• Tracked and evaluated

0.2 The problem with unstructured responses

Here’s what a typical unstructured response might look like:

The image shows a part with a potential defect. There's a small dark spot near the rim of the part that appears to be a void. The defect is approximately 15x15 pixels in size and seems to be located in the lower left quadrant of the image. It's not very pronounced but could be significant if it represents a manufacturing flaw.

This response is informative for a human, but it’s not machine-actionable. Parsing it reliably requires complex NLP and would be fragile to changes in the LLM’s output format.

0.3 Interactive comparison: Unstructured vs Structured

Drag the slider below to see the difference between unstructured text and structured JSON output:

The image shows a part with a potential defect. There's a small dark spot
near the rim of the part that appears to be a void. The defect is
approximately 15x15 pixels in size and seems to be located in the lower left
quadrant of the image. It's not very pronounced but could be significant if
it represents a manufacturing flaw.

{
  "defect_type": "void",
  "location": "lower_left_quadrant",
  "size_px": [15, 15],
  "severity": "low",
  "confidence": 0.62
}

left: unstructured text  |  right: structured JSON (illustrative sketch -- field names like defect_type/size_px are simplified for this comparison, not the real Defect schema below)

0.4 Getting structured output from LLMs

There are several approaches to get structured output from LLMs:

1. Prompt engineering - Carefully crafting prompts that encourage the LLM to output in a specific format
2. System prompts - Using system instructions to set the tone and format expectations
   
3. JSON schema specification - Explicitly defining the expected structure
4. Output parsing - Adding validation and error handling to catch malformed responses

0.5 Using JSON schema with LLMs

Let’s look at how we can structure our tool to return well-defined JSON:

Click to see structured JSON schema

{
  "defects": [
    {
      "bbox": [150, 200, 165, 215],
      "type": "void",
      "severity": "medium",
      "confidence": 0.85,
      "explanation": "A small dark circular region in the lower left quadrant that appears to be a void or gas pocket in the metal part."
    }
  ]
}

0.6 Interactive Schema Explorer

Below is an interactive tool that helps visualize how different JSON schemas can be structured for defect detection:

JSON Schema Playground

Try modifying the defect properties below:

BBox Coordinates:

Defect Type: Void Inclusion Crack Porosity Other

Severity: Low Medium High

Confidence: 0.85

0.7 The schema, for real

Traced from notebooks/agentic-zero-to-advanced/structured_output_demo.py – this is the actual Pydantic model the LLM’s response gets validated against, not a sketch:

notebooks/agentic-zero-to-advanced/structured_output_demo.py (schema)

from typing import List, Literal
from pydantic import BaseModel, Field

class Defect(BaseModel):
    bbox: List[int] = Field(description="[x1, y1, x2, y2] in pixels")
    type: Literal["void", "inclusion", "crack", "porosity", "other"]
    severity: Literal["low", "medium", "high"]
    confidence: float = Field(ge=0.0, le=1.0)
    explanation: str

class DefectReport(BaseModel):
    defects: List[Defect]

The Literal types matter more than they look like they should: they turn “the model made up a new category” into a hard validation failure instead of a silent, unnoticed drift in your data.

0.8 The real input: Part 2’s detector, not a raw image

The model here is Qwen3-Coder-30B, served locally via vLLM – a text model, no vision. It never sees the X-ray. What it gets is the real JSON from Part 3’s inspect_xray tool: the classical top-hat detector’s candidate boxes and their strength, with no semantic label attached. The LLM’s actual job is to turn “these pixel coordinates and this confidence number” into a defect type, a severity, and a human-readable explanation – which is a genuinely harder and more honest task than it sounds, because it has to admit when the numbers alone can’t tell it what kind of defect this is.

notebooks/agentic-zero-to-advanced/structured_output_demo.py (prompt)

SCHEMA_JSON = json.dumps(DefectReport.model_json_schema(), indent=2)

SYSTEM_PROMPT = f"""You are an industrial X-ray inspection assistant. You will be given the raw
output of a classical computer-vision void detector (candidate bounding boxes, each with a
strength, no semantic labels) for one X-ray image of a metal part. Turn it into a structured
defect report.

Output ONLY a single JSON object matching this schema -- no markdown fences, no prose before or
after it:

{SCHEMA_JSON}

Rules:
- Emit exactly one entry in "defects" per candidate box in the input, in the same order.
- Infer "type" and "severity" from the box's size, position, and the candidate's strength
  -- you do not have the image, only these numbers, so be conservative and say so in
  "explanation" when the numbers are genuinely ambiguous.
- If the detector reports zero candidate boxes, return {{"defects": []}} -- that is a valid,
  clean result, not a failure.
- Never invent a defect that is not backed by a candidate box in the input."""

{SCHEMA_JSON} is a plain f-string substitution (the variable, computed one line above from Pydantic’s own .model_json_schema(), so the prompt’s schema can never drift out of sync with the code that validates against it). {"defects": []} is the escaped-brace way to put a literal {"defects": []} into an f-string. If you’re copying this into your own prompt-templating code and it’s not an f-string (e.g. .format()), the escaping rules differ – check which one you’re actually using before assuming this pastes in unchanged.

0.9 Validating structured output, with a real retry – and a fix for the bug in this post’s first draft

An earlier draft of this exact section had this fallback:

except json.JSONDecodeError:
    return {"defects": []}

That line looks harmless and is actually the most dangerous line in the whole post: it makes “the model’s response was malformed” and “the part is genuinely defect-free” produce the identical output. Anything downstream that reads defects: [] can no longer tell “all clear” from “I have no idea what happened here” – exactly the failure mode a real inspection system cannot afford.

The fix actually implemented here: parse, validate against the Pydantic schema above, and if either step fails, retry once with the exact error message fed back to the model. If it still fails after the retry, return an explicit needs_review status with the raw response attached – never an empty list standing in for a failure.

notebooks/agentic-zero-to-advanced/structured_output_demo.py (validate + retry)

def parse_and_validate(raw_text: str) -> tuple[DefectReport | None, str | None]:
    """Returns (report, None) on success, (None, error_message) on failure."""
    try:
        data = json.loads(strip_fences(raw_text))
    except json.JSONDecodeError as exc:
        return None, f"JSONDecodeError: {exc}"
    try:
        return DefectReport(**data), None
    except ValidationError as exc:
        return None, f"ValidationError: {exc}"


def structured_report_for_image(client, image_id: str) -> dict:
    tool_result = inspect_xray(image_id)
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Detector output for {image_id}:\n{tool_result}"},
    ]

    raw = call_llm(client, messages)
    report, error = parse_and_validate(raw)
    if report is not None:
        return {"image": image_id, "status": "ok", "attempts": 1,
                "defects": [d.model_dump() for d in report.defects]}

    # One retry: feed the exact error back, ask for a fix, nothing else.
    messages += [
        {"role": "assistant", "content": raw},
        {"role": "user", "content": (
            f"That response failed validation with this error:\n{error}\n\n"
            "Return ONLY the corrected JSON object matching the schema above. "
            "No markdown fences, no explanation."
        )},
    ]
    raw_retry = call_llm(client, messages)
    report_retry, error_retry = parse_and_validate(raw_retry)
    if report_retry is not None:
        return {"image": image_id, "status": "recovered_after_retry", "attempts": 2,
                "first_attempt_error": error,
                "defects": [d.model_dump() for d in report_retry.defects]}

    return {"image": image_id, "status": "needs_review", "attempts": 2,
            "first_attempt_error": error, "second_attempt_error": error_retry,
            "raw_response": raw_retry[:800]}

0.10 The real result, measured on all 600 held-out images

Ran against Qwen3-Coder-30B (local vLLM, temperature=0.2) over every image in the held-out test split of the void dataset, not a subset. Six hundred calls, one per image:

Images	600
Valid on first try	600 / 600 (100%)
Recovered after retry	0
Needs review (both attempts failed)	0
Candidate boxes emitted	1,718
Count matches the tool (one entry per candidate)	600 / 600
Every bbox equals the tool’s box, in order	600 / 600
Images with zero candidates	141
… of which the model answered {"defects": []}	141 / 141

The honest headline: the retry path never fired. With a strict system prompt, temperature=0.2, and a small, capable coder model, malformed JSON did not happen across 600 real calls. The old version of this post could only say that about 72 calls, and it flagged one gap: it never exercised the “zero candidates, return an empty list” instruction, because every image had candidates. This dataset has 141 images where the detector raises nothing, and the model handled all of them correctly. That gap is closed.

What the run verified, and what matters going into a pipeline: every response kept the structure faithful to the input, down to the exact box coordinates. That is the property the rest of the series’ tools (the business rules, the report) depend on. But it is a copying task: the boxes are handed to the model, so getting them back unchanged says little about judgement.

Valid JSON is not the same claim as correct labels. The model never sees the image. It infers type and severity from a box’s size, position and strength. The Literal type guarantees every label is one of the five allowed strings, not that it is the right one. With exact ground truth I can finally check that. structured_semantic.py splits the 1,718 candidates into the 284 that match a real void and the 1,434 that are false alarms (the held-out numbers from Part 2) and compares what the model wrote:

type the model wrote	for real voids (284)	for false alarms (1,434)
void	282 (99%)	1,375 (96%)
porosity	2	50
other	0	7
inclusion / crack	0	2

The label barely depends on the truth: the model calls almost everything a void, real or not. That is partly my schema’s fault. The dataset has one defect class and the schema offers no “this is not a defect” option, so the field cannot carry the one distinction that matters. severity and confidence show a small signal at best:

	real voids	false alarms
severity low	126 (44%)	997 (70%)
severity medium or high	158 (56%)	437 (30%)

and confidence ranks a real void above a false alarm with an AUC of 0.645 (0.5 is a coin flip), against 0.622 for the detector’s own strength number, which the model was handed. So the model’s confidence is only marginally better than the number it was reading, and the confidence values cluster on a few round numbers (0.6 for 675 of the 1,718, 0.7 for 389), as you would expect from a model writing a plausible-looking number rather than computing one. I have no confidence interval on that 0.02 gap, so I would not claim it is real.

This is the lesson of the post in one table: perfect structure, no information. A downstream rule that read type == "void" would keep 96% of the false alarms.

Two caveats, stated plainly rather than glossed over:

• This is one model, one prompt, one temperature, on structured extraction from numbers it was already handed. It is not a general claim that “structured output never fails.” The retry and needs_review scaffolding stays in the code as a safety net, tested below, not because it was proven unnecessary.
• The dark mistake from Part 3 did not appear here (0 of 1,718 explanations call a candidate dark), most likely because this prompt asks for one defect entry per candidate rather than a free summary. I did not test why.

Full per-image output: notebooks/agentic-zero-to-advanced/out/structured_output_results.json; the analysis above: out/structured_semantic_summary.json.

0.10.1 Proving the untested paths actually work

Since the real run never hit recovered_after_retry or needs_review, those two branches were technically unexercised code – a safety net nobody had actually pulled on. So notebooks/agentic-zero-to-advanced/test_retry_paths.py injects known responses through a fake client with the same .chat.completions.create(...) interface aisuite uses (mocking only what the LLM would have said, not our own parsing/validation/retry logic), and runs the real structured_report_for_image function against them:

notebooks/agentic-zero-to-advanced/test_retry_paths.py (the three cases)

r1 = run_case("valid on first try", [VALID_JSON])
assert r1["status"] == "ok" and r1["attempts"] == 1

r2 = run_case("broken JSON first, valid on retry", [BROKEN_JSON, VALID_JSON])
assert r2["status"] == "recovered_after_retry" and r2["attempts"] == 2

r3 = run_case("fails validation both times", [INVALID_SCHEMA, BROKEN_JSON])
assert r3["status"] == "needs_review" and r3["attempts"] == 2
assert "raw_response" in r3

Real output, all three assertions passing:

=== Case 2: broken JSON first, valid on retry ===
{
  "status": "recovered_after_retry",
  "attempts": 2,
  "first_attempt_error": "JSONDecodeError: Expecting property name enclosed in double quotes...",
  "defects": [ ... ]
}

=== Case 3: fails validation both times -> needs_review ===
{
  "status": "needs_review",
  "attempts": 2,
  "first_attempt_error": "ValidationError: 4 validation errors for DefectReport\ndefects.0.type\n  Input should be 'void', 'inclusion', 'crack', 'porosity' or 'other' ...",
  "second_attempt_error": "JSONDecodeError: Expecting property name enclosed in double quotes...",
  "raw_response": "{ this is not valid json at all"
}

Now the safety net is tested, not just untried – both failure paths demonstrably do what the code claims, independent of whether the real 600-image run happened to need them.

0.11 What I did not do

• No fix for the useless label. I did not add a “not a defect” option to the schema and rerun. That is the obvious next step, and Part 7 sidesteps it by asking the model a different question.
• No other model, prompt or temperature. One setting, one run.
• No confidence interval on the AUCs or on the 99% / 96% label rates.
• No guided decoding (see below), so I cannot say how it compares.
• No check that the explanations are true. I checked structure, counts, boxes and labels, not whether each sentence of free text is accurate (Part 3 showed that it often is not).

0.12 Benefits of structured output

Structured output brings several benefits to our agentic system:

1. Reliability: Code can parse responses without complex NLP
2. Consistency: Predictable data formats across different LLM calls
3. Usability: Easy to store, compare, and analyze results
4. Integration: Can easily connect to databases, APIs, and other systems
5. Evaluation: Simple metrics can be computed on structured data

0.13 Common pitfalls with structured output

1. LLMs sometimes ignore instructions. Didn’t happen in this post’s 600-image run (see above), but plan for it anyway – the retry/needs_review scaffolding exists precisely because prompt compliance isn’t a guarantee, only an empirical result for one model/prompt/temperature combination, on one day, against one image set.
2. Schema mismatches: The LLM might not follow the exact schema you defined
3. Validation complexity: Need robust validation and error handling
4. Performance overhead: Validation adds computational cost

A natural question: since this is served through vLLM already, why not use vLLM’s own guided decoding (constrained generation that forces the output tokens to match a JSON schema grammar, so malformed JSON becomes structurally impossible rather than just unlikely) instead of prompt-and-retry? It’s a real, better answer for this exact setup – deliberately left out of this post’s own pipeline, though: prompt-plus-validation is the general-purpose pattern that also works against a cloud API with no grammar-constraint support, and the retry/needs_review path this post builds is what you fall back on for those cases regardless. Worth its own follow-up now that the prompt-only version has a real measured baseline to compare against.

0.14 Best practices for structured output

1. Start simple: Begin with a minimal schema and expand as needed
2. Use Pydantic: Leverage libraries like Pydantic for automatic validation
3. Include examples: Provide clear examples in your prompts
4. Add fallbacks: Have fallback mechanisms when validation fails
5. Test thoroughly: Test with various inputs to ensure robustness

0.15 What’s next

Part 5 kicks off Arc 2 — what makes a system truly “agentic”: planning, tool use, memory, and control flow. We’ll move beyond simple tool calling to build systems that can make decisions and iterate on their approach. Link goes live once that post is written.

0.16 Jargon, explained simply

Structured output. A model response constrained to a defined shape (JSON with specific fields, for example) rather than free-form prose, so downstream code can reliably parse it.

Daily-life version: a tax form with labeled boxes for name, income, and date, instead of a rambling free-text letter explaining your finances – the person (or program) on the other end can process a filled-in box far more reliably than prose.

Pydantic. A Python library for validating data structures using type hints, making it easy to define and validate complex data schemas.

Daily-life version: a recipe card that not only tells you what ingredients to use, but also validates that you have the right amount of each one before you start cooking.

JSON schema. A way to describe the structure of JSON data, specifying what fields are required, what types they should be, and what values they can take.

Daily-life version: a blueprint for a house that specifies exactly what materials to use, where each room should go, and how many windows and doors are required.

Validation. Checking that data conforms to expected rules and formats before accepting it as valid.

Daily-life version: checking that a driver’s license has the right photo, name, and ID number before letting someone drive.

0.17 Reproduce this yourself

Show reproduce commands

git clone https://github.com/HasanGoni/quarto_blog_hasan
cd quarto_blog_hasan/notebooks/agentic-zero-to-advanced
uv sync
cp .env.example .env   # edit: local vLLM (see below) or your own OPENAI_API_KEY
uv run python structured_output_demo.py && uv run python structured_semantic.py

Needs either a cloud OPENAI_API_KEY, or a local OpenAI-compatible server (this post’s run used vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct on localhost:8000, matching Part 3’s setup) with .env set to OPENAI_API_KEY=local and OPENAI_BASE_URL=http://localhost:8000/v1. Runs over the whole held-out split by default (600 images, AGENTIC_ZERO_MAX_IMAGES=N to cap it; on my shared GPU the full run took roughly half an hour) and writes per-image results plus a summary to out/structured_output_results.json / out/structured_output_summary.json. Then uv run python structured_semantic.py produces the label analysis. Build the dataset first, as described in Part 2.

0.18 Further reading

• LangChain Structured Outputs Documentation
• Pydantic Data Validation
• JSON Schema Specification