Images removed. This post’s figures showed frames from GDXray or outputs of models trained on them. GDXray is free for research and education only, and redistribution is not allowed, so the figures are gone. The text, code, numbers and plots are unchanged. The new synthetic void dataset is what this blog shows from now on.
This post is part of the Local-AI Defect Inspection Series. Full code: github.com/HasanGoni/lfm2-vl-xray-defects.
0.1 Where this picks up
Part 2 ended with local VLMs failing hard at zero-shot wafer-defect classification — 1-2/9, mostly by picking one default answer and riding it regardless of what was on screen. That result bugged me more than I expected it to. The obvious next question: does fine-tuning actually close that gap, or does a 450M model just learn a different, more sophisticated-looking shortcut?
I decided to push further than plain classification, partly because “defect or not” felt like too easy a target to declare victory on. Instead I wanted LFM2-VL-450M to say where — a bounding box, grounded against real annotation data, evaluated with real IoU, not a vibe check where I squint at a prediction and decide it looks close enough.
Short version: detection got measurably better. Localization never worked, in any of the four configurations I tried — and I can show you exactly what it did instead. I went in expecting a messier, more ambiguous result than this. What I got was cleaner and, honestly, a little more humbling.
0.2 The task
Input: one X-ray image + a fixed prompt. Output: a JSON object.
PROMPT = (
"Examine this X-ray image of a metal casting for defects (cracks, porosity, voids). "
'Respond with only a JSON object of the form {"defect": true|false, "boxes": [[x1,y1,x2,y2], ...]}. '
"Each box is a defect region in x1,y1,x2,y2 pixel coordinates normalized to a 0-1000 scale "
"relative to the image width and height. If there is no defect, use an empty list."
)Boxes are normalized 0-1000 relative to each image’s own width/height — a standard grounding convention, and necessary here since GDXray images vary in resolution. Schema, enforced at generation time with Outlines:
Outlines constrains decoding token-by-token so the model cannot emit malformed JSON — no retry loops, no regex parsing of free text. This is the kind of infrastructure choice that feels like overkill until the first time a model hands you {"defect": true, "boxes": [[12, 34, and you realize you’d otherwise be writing a janky recovery parser at 11pm.
One deliberate scope decision: no defect-type label (crack vs. porosity vs. shrinkage). GDXray doesn’t provide one — only bounding boxes. I was tempted, briefly, to invent plausible type labels just to make the task look richer for this post. I didn’t, because that would have broken the one thing that actually makes this series credible to me: every ground-truth label is real.
0.3 Dataset: GDXray Castings
GDXray+ Castings group: 2,727 X-ray images across 67 series, mostly aluminum automotive parts (wheels, knuckles), each with defect bounding-box annotations in a per-series BoundingBox.mat. Free for research/education use.
0.3.1 Obstacle #1: the official download is scripted-download-blocked
GDXray’s Dropbox links return a JS-gated interstitial to any non-browser client — curl with a spoofed user-agent, cookies, referer headers, none of it got past a Dropbox - Error page. This ate more of my afternoon than I’d like to admit. I found a Kaggle mirror instead (samarthgoel2604/gdxray-without-synthetic-for-defectd) that preserves the original per-series folder structure, pulled via kagglehub:
0.3.2 Obstacle #2: the bounding-box column order is not what you’d guess
BoundingBox.mat has one row per defect region: 5 numbers. I assumed [image_index, x1, y1, x2, y2] — the “obvious” convention, because it’s the one I’ve seen everywhere else. Wrong. Cross-checking against a reference implementation (maxkferg/metal-defect-detection), the actual column order is [image_index, x1, x2, y1, y2] — x-range then y-range, not interleaved. My first guess would have silently produced a degenerate box (x2 < x1) for at least one real annotation I checked by hand — the kind of bug that doesn’t crash, doesn’t warn you, and just quietly trains the model on garbage. Small thing, but it’s exactly the sort of mistake that would have invalidated everything downstream without ever throwing an error.
I didn’t just trust the reference code either — I rendered the corrected box on a real image, because at this point I trust my own eyes more than either source:
[Figure removed: it showed GDXray pixels, or a model output made from them, and the GDXray licence does not allow redistribution.]
Lesson I keep re-learning: when a dataset format is undocumented and a reference implementation disagrees with the “obvious” reading, render a box on a real image before trusting either.
0.3.3 Manifest and split
Ground truth gives a clean per-image label for free: an image is defect if its index appears in any row of its series’ BoundingBox.mat, no_defect otherwise.
87 series, 3,981 images, 714 defect (17.9%) / 3,267 no_defect (82.1%) — a real, imbalanced, single-label split close to real-world defect rates. 1,319 total boxes; 318 images have more than one defect region.
Split train/eval by series, not by image — images within one series are highly correlated (same casting, adjacent rotation angles), so a naive image-level split would leak information between train and eval and quietly flatter every number in this post. I was paranoid enough about this to check it twice before running anything. Result: 3,313 train images (70 series) / 668 eval images (17 series), same seed reused across every experiment below so comparisons stay fair.
0.4 Zero-shot baseline
Before touching any weights, I ran the same prompt against the vanilla model on all 668 eval images.
| accuracy | precision | recall | mean IoU | |
|---|---|---|---|---|
| Zero-shot | 26.3% | 21.0% | 100% | 0.00 |
100% recall sounds great for about as long as it takes to open the confusion matrix: the model predicted defect: true for 623 of 668 images (93.3%) — it wasn’t detecting defects, it was just saying yes to nearly everything, which is a strategy but not a useful one. And on the 131 true positives, mean IoU against ground truth was exactly 0.00 — zero real localization signal, zero-shot. Not a great start, but at least an honest one.
(Side note, and this one genuinely amused me: a differently-worded plain-text prompt asking only for defect/no_defect — no JSON, no boxes — produced the opposite degenerate behavior: no_defect on all 668 images, 0% recall. Prompt framing alone flipped which majority-class shortcut the model defaulted to. Neither framing engaged with the actual image content zero-shot — the model just picked whichever lazy answer the prompt nudged it toward.)
0.5 LoRA fine-tune
0.5.1 Config
Following Liquid AI’s own fine-tuning guidance: rank 16, lora_alpha=32 (2×r). Their docs list generic target-module names for their model family (q_proj, v_proj, fc1, fc2, linear, gate_proj, up_proj, down_proj) — but half of those don’t exist as literal module names in LFM2-VL-450M’s actual architecture, which I only found out because I checked instead of copy-pasting. I checked with named_modules() before trusting the doc:
0.5.2 Training target = the same schema Outlines will constrain at eval time
Training the model to emit exactly what Outlines will later constrain it to, rather than free-text, so there’s no train/eval mismatch in output format. Felt obvious once I’d written it, less obvious before — my first draft of this pipeline trained on loosely-formatted text and I had to backtrack.
0.5.3 Label masking via prompt-prefix trick
Loss should only apply to the response tokens, not the image+prompt context. Rather than parsing the chat template’s special-token structure by hand — which I really did not want to do — I exploited that the tokenized prompt is an exact prefix of the tokenized full conversation (image processing is deterministic regardless of what text follows it). I verified this holds before relying on it, because “should be true” and “is true in this specific tokenizer” are not the same claim:
prompt_inputs = processor(images=[img], text=prompt_text, return_tensors="pt")
full_inputs = processor(images=[img], text=full_text, return_tensors="pt")
prompt_len = prompt_inputs["input_ids"].shape[1]
labels = full_inputs["input_ids"].clone()
labels[:, :prompt_len] = -100 # ignored by cross-entropy0.5.4 Class-weighted sampling
Train split is 17.6% defect. Left alone, a fine-tune on this data will happily relearn “always say no_defect” — it’s an easy 82% accuracy, and I’ve watched a model take that shortcut before, so I wasn’t going to let it happen twice. Inverse-frequency weighted sampling forces roughly balanced defect/no_defect exposure per epoch:
0.5.5 Obstacle #3: training instability
First run (flat lr=1e-4, no schedule, 800 steps, checkpoint every 200): loss trended down (1.23 → 0.56), which looked promising, right up until quick-eval on a held-back 80-image subset oscillated between collapsing to “always no_defect” and a more balanced state, checkpoint to checkpoint:
| step | accuracy | recall | precision |
|---|---|---|---|
| 0 (baseline) | 52.5% | 100% | 51.3% |
| 200 | 50.0% | 0% | — collapsed |
| 400 | 60.0% | 52.5% | 61.8% |
| 600 | 50.0% | 0% | — collapsed again |
| 800 (final) | 48.75% | 12.5% | 45.5% |
Watching recall bounce from 52.5% to 0% to 52.5% to 0% again across a single training run is not a fun way to spend an evening. I added a warmup + cosine LR decay schedule (peak lr=5e-5, 10% warmup) and reran, checking every 100 steps instead of 200, half-expecting that to fix it. It didn’t fix the oscillation — recall still bounced 0%→100%→12.5%→0%→32.5%→30% across the run. That ruled out “missing LR schedule” as the root cause and points instead at the small effective batch size (gradient-accumulated batch of 4, on a dataset with only ~583 real defect training examples after oversampling) producing a high-variance gradient estimate. Something I’d want to fix with a larger effective batch before trusting this recipe further — I’m noting it here rather than quietly smoothing it over, because I’d want to know this if I were reading someone else’s post.
0.5.6 Results — full 668-image eval, four configurations
| config | accuracy | precision | recall | mean IoU |
|---|---|---|---|---|
| Zero-shot | 26.3% | 21.0% | 100% | 0.00 |
| LoRA, flat lr, step 400 | 60.3% | 26.6% | 58.0% | 0.00 |
| LoRA, flat lr, step 800 (final) | 75.6% | 26.5% | 13.7% | 0.00 |
| LoRA, warmup+cosine, step 800 (final) | 66.0% | 23.9% | 33.6% | 0.00 |
Detection genuinely improved, and this is the part I’m actually pleased with. Zero-shot’s degenerate “flag everything” (100% recall / 21% precision) became a real, much more balanced decision at step 400 — 58% recall at similar precision. Unstable across configurations, not a clean monotonic win — I’d be lying if I called this recipe “solved” — but real signal that the model learned something about defect presence.
Localization never moved. At all. In any configuration. Mean IoU is exactly 0.00 across zero-shot and all three fine-tuned checkpoints. Four separate runs, same dead number every time.
0.5.7 I didn’t just report 0.00 — I checked why
A number that’s exactly 0.00 in four separate runs is either a very clean finding or a bug in my eval code, and I’ve been burned by the second option enough times to not just trust the first. Before writing this up, I pulled the raw predictions:
true: [[456.0, 502.0, 489.0, 545.0]] pred: [[0,0,0,0], [0,0,0,0], [0,0,0,0], [0,0,0,0]]
true: [[459.0, 483.0, 489.0, 524.0]] pred: [[0,0,0,0], [0,0,0,0], [0,0,0,0], [0,0,0,0]]
true: [[462.0, 465.0, 492.0, 503.0]] pred: [[0,0,0,0], [0,0,0,0]]
true: [[466.0, 447.0, 499.0, 483.0]] pred: [[0,0,0,0], [0,0,0,0], [0,0,0,0], [0,0,0,0]]Every single predicted box, on every image, is [0,0,0,0] — regardless of where the real defect actually is. That’s a genuinely funny (in a bleak way) thing to discover after a training run: the fine-tuned model learned to say "defect": true more accurately, then just… gave up on the harder task hiding inside the same JSON object, and found the laziest possible way to make the coordinate loss go down. [0,0,0,0] is a cheap, constant way to reduce the coordinate-token loss without ever learning to regress a real value through next-token prediction — and apparently a local optimum the training found and settled into, every time.
That’s a much stronger, more specific finding than “IoU was low,” and honestly a more satisfying one to have dug up than to have just reported the number and moved on. Detecting that something’s wrong turned out to be learnable in an 800-step LoRA budget on a 450M model. Detecting where — asking a generalist small VLM to learn numeric grounding purely through text-token generation, from a few hundred examples — did not work once, in four attempts.
0.6 What’s next
I don’t think a 450M generalist VLM, fine-tuned this way, is the right tool for pixel-level localization from scratch, and I say that having genuinely wanted it to work. The natural next experiment: split the job. Use a dedicated segmentation/prompted-detection model (SAM3, or similar) to find candidate defect regions, and hand that result to the VLM for reasoning — instead of asking one small model to learn both “what” and “where” end-to-end through the same text-generation objective.
If you’ve combined a segmentation model with a small VLM this way for a narrow visual task, I’d genuinely like to hear about it — curious whether the extra moving part is worth it in practice, or just trades one set of headaches for another.
Full code, all scripts referenced above, and the raw per-image results: github.com/HasanGoni/lfm2-vl-xray-defects
Reuse
Citation
@online{goni2026,
author = {Goni, Hasan},
title = {Fine-Tuning {LFM2-VL-450M} for {X-ray} {Defect} {Detection}
and {Localization}},
date = {2026-08-09},
langid = {en}
}