WilD3DGen · Debugging the v2.3 3D-captioning regression

0 · The symptom

Stage-1 v2.3 (Qwen3.5-2B + 1536 discrete 3D codes, 3 epochs over 1.87M assets) came out of training with an ability its predecessor had: it could no longer say what a 3D asset is. Fed the ground-truth codes for a rifle, v2.2 answered "a Star Wars stormtrooper" for a stormtrooper and named objects correctly in general; v2.3 wrote fluent, well-structured 200-word prose about an "anthropomorphic fox". An F1 car came back as a "cargo ship". The prose was better than v2.2's. The identity was invented.

This note is the debugging trace: four hypotheses, three of which were wrong and one of which was wrong in an instructive way, plus an eval-hygiene bug found along the way that turned out to matter more than the original question.

1 · First hypothesis: identity was diluted — measured, and refined

The obvious suspect was a change I had made: the [3D Und] target went from v2.2's 32-word identity-dense caption to a 200-word structural recap. Cross-entropy averages over tokens, so identity is ~3 words of 200 instead of a large share of 42.

The first measurement refuted this: teacher-forced NLL on the first sentence was 0.839 vs 0.744 on the rest — a ratio of 1.13×, nothing. But that measurement repeated the very mistake it was testing: the recap's first sentence is ~20 words, of which identity is 2–3, so the average buried the signal. Per-token the picture is unambiguous.

2 · The decisive test: ablate the codes

Low loss on the prose is ambiguous — it could mean the codes make it easy, or that the model is reciting a language prior and never reads the codes at all. Those predict opposite fixes, so: re-run the same NLL with the code tokens destroyed (shuffled within family, and replaced by a single constant), keeping the prompt shape identical.

So the codes enter the caption almost entirely through the first few tokens. Expressed as information, the whole 1536-code sequence buys ln(32.3 / 13.3) = 0.89 nats on the identity decision. Running the identical probe on v2.2 — its own checkpoint, its own 512-code cache, its own data — gives ln(145.9 / 2.7) = 3.99 nats.

3 · Why it collapsed: a shortcut at the one position that matters

The constant column is the tell. Without codes, v2.2 is at ppl 146 — it genuinely cannot guess. v2.3 is at ppl 32 — it can. That difference is a property of the text, so it is checkable on CPU with no model at all. Measuring the prior entropy of the opening two words over 1.67M captions:

Mechanism: the recap was written as a structural description, so it opens with shape adjectives — "A stylized…", "A flat, elongated…". Two in five start with the same two words. A language prior therefore answers the one position where identity is decided, the code→identity pathway is never required to reduce loss, and it does not develop. Classic shortcut learning. At generation time the model commits to a wrong noun at position 1 and then elaborates coherently on that wrong commitment for 200 words — which is exactly why the output looks confidently insane rather than merely poor.

Two hypotheses died here and are worth recording. (a) The recap isn't diverse enough — false: whole-caption vocabulary is 13,025 vs v2.2's 4,252 and boilerplate is 2.2% vs 3.1%. It is more diverse overall and stereotyped only at the opening. (b) The und share collapsed — true but secondary: 17.8% → 6.2%, because the export subsamples und with #320000 while everything else grew 4×. Raw jsonl line counts say the share barely moved; they are not what training sees.

4 · The fix, and what one epoch bought

Revert the und target to v2.2's 32-word caption, raise its share 6.2% → 28%, and continue-train from checkpoint-9808 — 711,714 samples, 2 nodes, ~85 min. Acceptance was deliberately not loss (the mixture changed, so loss is not comparable) but three behavioural checks.

Real but incomplete, and the reason is visible in the same measurement: with codes destroyed the identity token still sits at ppl 32 rather than v2.2's 146. Three epochs of recap training baked in the "a stylized / sleek…" opening habit and one epoch of the corrected target does not erase it. Two different cars still produce verbatim identical und text — the readout is category-level, not instance-level. Closing the rest needs a longer or fresh run, not another recipe tweak.

5 · The bug that mattered more: the eval set had been swallowed

Mid-investigation, a sanity check on the 22-asset held-out set: 19 of them are inside v2.3's 1.67M pretrain pool. The seen/unseen column was computed against v2.2's training list (856k) and silently stopped meaning anything when the pool grew. Nothing errored. The numbers kept printing.

5.1 · What it invalidated, and what it flipped

• v2.3's headline "0.034 → 0.159, a 4.7× gain" is a training-set number. True held-out is 0.089 mean / 0.042 median, i.e. ~1.5–2× over the previously known 0.04–0.06 — and 45% of the SS tokenizer's own 0.197 reconstruction ceiling.
• The und identity failures are worse than they looked: the model could not name objects it had trained on.
• It reversed a design decision. On the contaminated set, text→3D CoT measured Δ −0.028 (t = −1.00) and the plan was to cut it from 70% to 30% of samples. On the clean set it is Δ +0.020 with t = +5.30. Paired A/Bs on a contaminated set are not automatically safe: the arms can generalise differently.

Fixed so it cannot recur: ROLLOUT_MANIFEST / ROLLOUT_TRAINSET across the three eval scripts, a new 300-asset set at manifests/heldout_v23/ (drawn from ~200k pool-external assets, each with an SS latent, a good-band render and a caption ladder), and an unconditional banner on every run:

[eval] 300 assets vs trainset pretrain_clean.jsonl (1,667,854)
[eval] !!! CONTAMINATED: 19/22 (86%) of the eval assets are IN the train set
       — this measures fit, not generalisation

6 · So what is the long caption actually for?

With a clean set the same question could finally be asked properly: does writing a 200-word description before emitting codes help generation? Answer, on 300 unseen assets, greedy:

6.1 · It is a rescue mechanism, not a quality multiplier

The band split is the mechanism. On assets the direct arm already handles, writing a caption first hurts (−0.014 image, −0.023 text): the model already knows what it is making, and 200 words of self-generated prose only adds one more chance to commit to something slightly wrong. On assets the direct arm fails outright, it is the whole difference between nothing and something. Net benefit is therefore failure rate × rescue size — which is why text→3D (67% base failure) wins decisively and image→3D (39%) comes out a wash.

Is it just that a 3-word prompt is too short? Partly, but that is not the mechanism.

6.2 · Why image→3D gets nothing: its own caption is wrong 38% of the time

The CoT text the model writes is itself a generation, so it can be scored. Dumping it for all 300 assets and checking whether it names the ground-truth object:

text→3D is handed the object in its prompt, so its caption is essentially clean signal and the rescue effect survives intact. image→3D has to infer identity from the render, gets it wrong in nearly two of five cases, and those cases inject a confidently wrong identity that the following 512 codes are then conditioned on. The rescue gains and the misidentification harms cancel — which is precisely the wash the paired test reports.

6.3 · And 182 of those 200 words are free-riding

If the mechanism is "commit to a global shape before emitting codes", the commitment lives in the first sentence — what it is and roughly what proportions — and the remaining 180 words describe edges and finish, which the geometry codes do not encode. Testable without retraining: take the model's own CoT text, truncate it, force the code block to start, and generate.

Full vs one sentence: Δ +0.003, t = +0.80 — nothing, on n = 256 paired. The mean nudges up with length while the median and the failure rate move the other way, which is the signature of no real effect. One sentence is at least as good on the metric that matters most (does anything usable come out at all).

7 · Final recipe

8 · Engineering footguns hit along the way

Four separate silent failures cost more wall-clock than the science did. All four share a shape: a default that was correct when written, expiring as the system around it grew.

• Stale tokenizer default. The rollout script defaults to v2.2's SS tokenizer. Same grid, same codebook size, no error — v2.3 codes decoded against it score 0.006 instead of 0.159. Now always passed explicitly and echoed in the log.
• Stale train-set default. The contamination above. Now printed every run.
• The GPU-keeper race. Idle held nodes run a managed vLLM service to hold utilisation; it stands aside for real work, but not instantly, and training allocates on startup — so a launch into a just-cleared node OOMs at ~29 GB/GPU of residual. Fix: park a 1 GB claim on every GPU first, poll until free, then launch. Misreading this once cost a 2× slower single-node run.
• Waiting only for success. A munge outage killed a run at step 295/597; the watcher was written to wait for a completion string, so it waited three hours. A second run lost two of three arms when a day-old allocation expired. Watchers now detect death (log mtime stalling, Slurm step disappearing) as well as completion, and checkpoints dropped from every 500 steps to every 150.

9 · Where this leaves the project

The caption recipe is settled and the remaining headroom in it is small. The discrete-token path generalises to 45% of its own tokenizer's ceiling and sits 4.5× below the conditioning path that does not route geometry through tokens at all. The bottleneck named by the earlier grounding probe still stands: codes are arbitrary symbols to the LLM, cross-entropy penalises a geometric neighbour exactly as much as an unrelated code, and RoPE never writes position into the residual stream. That is where the next effort belongs — codebook-tied embeddings plus a geometric auxiliary loss, or letting the VLM emit conditioning only and leaving geometry to the flow — not in further data-recipe tuning.

10 · What to do next, in order

1. Shorten the CoT target to the recap's first sentence and rebuild the generation arms. Small, mechanical change to the data builder; §6.3 says it costs nothing in quality and returns ~90% of the CoT token budget plus a ~5x faster CoT step. Do this first because everything after it trains faster.
2. Train the image-side CoT arm against the VQA-quality description. On the same renders, the [VQA] path identifies objects markedly better than the [3D Gen … CoT] path does (62% object accuracy, §6.2), which is the whole reason image→3D CoT nets out to a wash. Lift that number and the arm should move from neutral to positive without any change to the architecture.
3. Then, and only then, the token interface. The recipe work is finished and its remaining headroom is a few percent. The measured gap — 0.089 generalisation against a 0.197 tokenizer ceiling and a 0.403 conditioning path — is not a data-recipe gap. The candidates are codebook-tied embeddings plus a geometric auxiliary loss (so that a near-miss code is penalised less than an unrelated one), or dropping discrete geometry emission entirely and letting the VLM supply conditioning only.