← Back to project

Mode collapse in a discrete-token 3D VLM — the diagnosis, and the fix

Why image→3D generated the same box for every input, how we proved the cause, and the recipe that fixed it · 2026-07-03
stage-13d-tokensmode-collapsediagnosistokenizerdecoding

Our Stage-1 VLM (Qwen3.5-2B + discrete 3D tokens) learned to read 3D well, but when asked to generate it, every input collapsed to roughly the same blobby shape — a torus became a bottle, a gear a smooth disc. This is the record of running that failure down to a single measurable cause and reversing it: shape and color now bind to the input, and a quantitative image-usage signal tripled.

One-line result. The collapse was over-determined by greedy decoding + too many tokens (5824/asset let the model satisfy the loss without ever looking at the image). Fix = sample (never greedy) + shorter codes (1536/asset) + input-noising + light CFG. The image-usage GAP went from a flat 0.04 to 0.12 bits/token and is still rising.
Read this first — what "generation" means here. At deployment our 3D VLM works like the base Qwen: image + text → understanding → hidden states, and a separate flow head reads those hidden states to generate 3D. The VLM never emits discrete 3D tokens at inference (this is not the BLIP3o-NEXT emit-a-blueprint design). So the discrete tokens are a training-only representation-shaping signal, and decoding the emitted tokens into a mesh — the "collapse" studied below — is a DIAGNOSTIC (a proxy for "did the hidden states learn 3D"), not the product path. We chased the collapse because a healthy diagnostic means healthy hidden states; the actual generator is the Stage-2 flow head.

1 · The symptom

Free-generating 3D from an image produced coherent-but-wrong volumes, identical in character across very different inputs. Greedy decoding was the worst case; even with sampling the shapes were generic.

v1 collapse: torus, gear, crate all decode to flat discs and slabs
The disease (v1, 5824-token model, greedy). Left = input / ground-truth mesh; right = model generation. Torus → flat disc, gear → slab, crate → rough box. The frozen decoder is fine — GT codes decode cleanly (the GT panels are correct); the model is emitting codes for "an average object".

2 · The diagnosis — four tests, one culprit

We resisted guessing and instead ruled hypotheses in or out with cheap measurements.

#HypothesisTestResultVerdict
1Token stream is low-information → CE rewards copying (the FAST failure mode)Per-token entropy + repeat-run audit of the VQ codesSS/shape/tex carry 10.8–11.9 bits/token, codebook fully used, no pathological repeatsRULED OUT
2The VQ decoder is brittle to imperfect (off-manifold) predicted codesDecode GT codes vs predicted codesGT decodes clean; predicted decodes to clean but wrong shapes, not garbage — decoder is faithfulRULED OUT
3The model ignores the conditioning image (condition-ignoring / posterior collapse)Image-usage GAP: teacher-forced CE on codes with vs without the imageGAP ≈ 0.04 bits/token, and flat across epochs — the image barely changes the predictionCONFIRMED
4It's just a bad decoding strategy (greedy)Greedy vs temperature-sampling on the same weightsSampling removes the degenerate loops/flats; still generic without more structureCONTRIBUTING
The smoking gun (Test 3). A text-conditioned probe (rich caption vs terse prompt, no image at all) gave the same ~0.04 GAP — so it isn't a vision-pathway problem, it's every condition being bypassed. With 5824 tokens of local code grammar available as an "easy path", the model minimizes CE by modeling the codes' internal structure and dilutes the image's contribution to near zero. That is exactly your intuition: too many tokens → the task is learnable without the condition.

Why the literature agreed before we changed anything

Three independent lines converge on the same architecture, which framed the fix and the roadmap:

SourceWhat it says for our case
Image-AR (LlamaGen, VAR, Chameleon, Emu3, Janus)Never greedy — every shipped model samples + uses classifier-free guidance. Greedy on long high-entropy streams is the textbook collapse setting.
MolmoAct2 (action pretraining)Same discrete-token-VLM problem. Uses 10% teacher-forced input-noising on structured codes + response-token dropout; final policy uses a flow expert, not raw token decode.
Kyvo / EVA01 / BLIP3o-NEXTDiscrete AR over 3D tokens works for structure; every high-fidelity textured generator uses continuous latents + flow. Discrete = blueprint, flow = fine geometry.

3 · The fix — v2 recipe

Everything below stacks; each item targets a specific finding above.

ChangeTargetsDetail
Shorter codes 5824 → 1536Test 3 (easy path)Re-trained 3 VQ tokenizers to 512 tokens each (SS/shape/tex). Fewer tokens → each carries more of the answer → the condition becomes necessary, not optional.
Input-noising 0.15Test 3 + exposure biasCorrupt 15% of teacher-forced code inputs (labels intact) — breaks the local-grammar shortcut, forces gradient toward the condition. (MolmoAct2 mechanism.)
Sampling + constrained grammarTest 4Never greedy; section markers injected deterministically + per-section vocabulary masking → runaway loops become impossible.
Condition-dropout 10% → real CFGgeneric outputs10% of image→3D samples trained image-free, so the unconditional branch is in-distribution and CFG actually works at inference.

4 · Results

0.04 → 0.12
image-usage GAP, SS section (bits/token) — 3× and still climbing
3 / 3
3D→text understanding correct (object + color + shape)
92%
VQA vs 83% base — no catastrophic forgetting
3.8×
shorter sequences → faster epochs, denser condition gradient

The GAP curve — binding emerging with training

v1's GAP was flat (the disease); v2's climbs monotonically and hasn't saturated at 10 epochs — the model keeps learning to use the image.

0.140.1050.070.0350 1.5k3k4.5k9.8k (10ep) training step → v1 baseline · 0.04 (flat = the disease) v2 ALL · 0.09 v2 SS · 0.12 GAP (bits/token)
Higher = model relies more on the image. v1 (red, dashed) never moved off 0.04; v2 (blue/green) climbs across the whole run.

Generation — from collapse to faithful

v2 plain generation, faithful
v2 · plain (sampling + constrained, no CFG). Clean surfaces; torus/crate faithful, gear teeth shallow.
v2 CFG generation, faithful with structure
v2 · CFG 1.5 (low guidance). Recovers gear teeth + center hole; color and topology bind to the input.
Faithful, coarse, on-purpose. Torus keeps its hole, gear grows teeth + a center hole, the wooden shed keeps brown walls and a green roof — the exact failures of v1, reversed. Low CFG (~1.5) beats high (3.5–5 fragments the surface). This is the Stage-1 diagnostic target: a healthy decode proxy means the hidden states carry geometry — fine generation is the Stage-2 flow head reading those hidden states, not this decode.

Understanding was preserved

5 · What we learned

Next. (1) v2.2 — clean up task-tag semantics ([VQA] = all non-3D tasks) — done; all three tags route cleanly. (2) Stage-2 — the actual generator: a flow head that reads the VLM's hidden states (read-once cross-attn, the winning cond arm) → SLAT → mesh. At deploy the VLM only does understanding (image+text → hidden states); it never emits 3D tokens — the discrete tokens stay training-only, for representation-shaping + reading + editing. (3) Editing is the payoff of a shared read/write vocabulary.
Qwen3.5-2B · 4×8 H200 · discrete 3D tokens on frozen TRELLIS.2 SLAT · best checkpoint stage1_v2/…/checkpoint-9790 (10 epochs, 1536 codes/asset). Full paths, scripts, and run instructions in the repo README at /fsx/home/weikai.huang/3dgen/vlm3d_stage1/README.md.