Root cause found: qwen's RoPE never writes position into the residual stream — position decodability is flat across all layers (linear R² ~0.52–0.59 vs DINOv3 0.94). The flow can't tell where a patch token came from.
Two working fixes: ADD-fusing real DINO features onto qwen tokens hits 0.380/0.327 IoU (≈ fusion, half the cond length) at just 2k steps; a single constant position-stamp table lifts the prod qwen-only plateau 0.256 → 0.319 (+25%).
What does NOT work: distilling the fusion teacher's velocities into a qwen-only student (KD) gains nothing — information absent from the student's input cannot be mimicked. And DINO-distilling the VLM improves its hiddens 2.5× without changing AR rollout at all — hidden quality is consumed by flow conditioning, not AR generation. Final verdict: after flow adaptation the internalized VLM lands at 0.287–0.292 — below the 0.319 stamp baseline; probe gains did not transfer (§5b).
NEGATIVE — probe decodability (R² 0.657) did not transfer to conditioning utility; representation shift (cos 0.30 vs old hiddens) disrupted what the flow exploited; 10k-adaptation caveat noted
Row order follows the story: reference → the problem → dead end → position-only fixes → content fixes → deployed fix → internalization (negative).
1 · Problem: qwen-only conditioning plateaus far below DINO-fused
SS (sparse-structure) flow conditioned on qwen-VLM hiddens alone plateaued at IoU 0.256 — flat for 10k steps on 425k assets. Not a data or compute problem.
Same model, same data, DINO-fused conditioning reaches 0.403 — a 1.6× gap attributable purely to the conditioning signal.
VLM AR rollout (discrete 3D tokens) is even worse, and far below its own tokenizer ceiling.
Conditioning path
SS IoU@64
Note
qwen-only hiddens (prod mech)
0.256
plateau — flat 10k steps, 425k assets
DINO-fused hiddens
0.403
same data / steps / flow
VLM AR rollout (discrete tokens)
0.04–0.06
discrete-tokenizer ceiling is 0.197
2 · Diagnosis: probes show the VLM is position-blind (and mostly content-lossy)
Position probe — can a linear probe decode each patch token's own (row, col) from the hidden state?
Representation
Linear probe R²
Mean position error (/32 patches)
DINOv3 patch features
0.941
1.68
qwen layer 1
0.535
~4.2–4.8
qwen layer 8
0.518
qwen layer 17
0.593
qwen layer 24
0.554
qwen is FLAT across all layers: RoPE rotates queries/keys inside attention but never writes position into the residual stream that the connector reads.
MLP probe (nonlinear): DINO 0.818 vs qwen best 0.717 — position is partially recoverable nonlinearly, but localization is still ~2× worse.
Content probe — ridge R² of qwen hidden → same-position DINO feature = 0.258: only ~26% of local appearance is retained per token.
Input pipeline exonerated: VQA on the same inputs is perfect, pixel stats clean, and the extractor matches the prod cache exactly (cos 1.0000).
The flow receives tokens that neither say where they are (address) nor fully what is there (content). Position is the cheap fix; content is the deep one.
Ablation IoU curves (left) and position-probe bars (right). The probe bars are the diagnosis: qwen position R² is flat ~0.52–0.59 at every depth while DINOv3 sits at 0.94 — the residual stream simply never carries the address.
3 · Six-arm conditioning ablation: what actually closes the gap
Setup: 16k assets, 8k steps, TRELLIS-pretrained init; IoU@64 on 13 held-out assets (a near-duplicate twin excluded); reported mean/median.
Arm
Mechanism
IoU@64 mean / med
Note
cross
last-layer read-once cross-attn (prod mech)
0.202 / 0.123
baseline
crosspos
+ generic 2D sincos, pre-connector
0.273 / 0.156
slow-burn — needs 2–4k steps before the flow exploits it
crossdpos
+ DINO-native position signature (content-averaged DINO features), post-connector
0.264 / 0.186
instant adoption already at 2k steps
plkv
8-layer per-layer-KV taps
0.235 / 0.169 (12k steps)
real but small — zero-init gates open to only ~10–16%
crossdadd
real per-asset DINO features ADD-fused onto qwen tokens (cond length 1045 vs concat-fusion 2074)
fusion-teacher velocity distillation into qwen-only student
0.249 (vs 0.264 warm-start)
NO GAIN — kd residual floors at 0.0186
crossdadd: 0.380/0.327 at 2k steps ≈ fusion quality at half the cond length. Adding DINO features directly onto qwen tokens works — the two signals coexist in superposition with negligible interference.
KD is a dead end: a student whose input lacks the spatial information cannot mimic a teacher that has it. Mimicry cannot create information.
Full-chain 3D renders, arms vs GT. crossdadd reconstructions track GT layout; the plain cross arm loses object placement and proportions.SS-only occupancy meshes — the most diagnostic view. Position-augmented arms recover coarse geometry where the position-blind baseline smears or drops structure.
4 · Prod fix: the DinoPosStamp (+25% from one constant table)
Continue-train from the 0.256 plateau with the DinoPosStamp: a constant 1024×1024 lookup table added post-connector at the image-token span. No new encoder, no extra tokens, no per-asset compute.
Checkpoint
IoU@64 mean
IoU@64 med
plateau (before)
0.256
0.129
+2k steps
0.291
0.16–0.19
final
0.319 (+25%)
fusion arm (control)
0.397–0.400
unaffected — stable across 7 measurements
0.256 → 0.319 (+25%) from a single constant lookup table. Position really was the address problem the probes said it was.
PROD before/after the position stamp: same assets, same flow — the stamped run recovers placement and coarse shape the plateaued run had lost.
5 · Internalization: DINO-distilling the VLM itself (OLA-VLM-style, REPA-shaped)
Since KD at the flow failed, fix the content problem at the VLM: v2.2 continue-train (1 epoch, 3h, 16 GPU) with aux heads on the last layer + layer 12 regressing frozen DINOv3 patch features per position; loss (1−cos) + 0.2·SmoothL1, weight 0.1.
Gates all green at every checkpoint:
Gate
Before
After
Verdict
held-out content probe R² (hidden → DINO)
0.258
0.657
2.5× — green
3D-understanding NLL
0.582
0.502
improved — green
rollout format validity
2/3
3/3
improved — a real distillation win
VQA task-mode (code-spew rate)
1/3
2/3
caveat: task-mode drift
AR rollout IoU (n=6)
0.040
0.039
unchanged — dissociation
Dissociation result: hiddens got 2.5× more DINO-decodable yet AR rollout is bit-identical in quality — hidden quality is consumed by flow conditioning, not by AR generation. The two pathways are decoupled.
Flow adaptation with rebuilt caches: step-0 (unadapted) sanity qwen 0.149 / fusion 0.397 — the drop is the expected distribution shift, fusion control intact. Final adapted verdict in §5b below.
KD negative result: the distilled student (0.249) does not even recover its own warm-start (0.264). The teacher's velocities encode spatial information the student's input never contained.AR rollout before/after DINO distillation — visually and numerically unchanged (0.040 → 0.039). The dissociation evidence: better hiddens feed the flow, not the AR head.
5b · Flow adaptation verdict: probe gains did not transfer (NEGATIVE)
Flow continue-trained on rebuilt caches from the internalized VLM; evaluated at ckpt-2000/4000/6000; training killed at 6k by protocol (flat trend below baseline).
Checkpoint (flow adapted on internalized-VLM caches)
qwen-only IoU@64
fusion control
step-0 (unadapted sanity)
0.149
0.397
ckpt-2000 / 4000 / 6000
0.287–0.292 (flat)
0.398–0.407 (intact)
baseline: old VLM + DinoPosStamp
0.319
0.397–0.400
NEGATIVE: probe decodability (R² 0.258→0.657) did not transfer to conditioning utility — the adapted flow lands at 0.287–0.292, BELOW the 0.319 old-VLM+stamp baseline. Representation shift (cos 0.30 vs old hiddens) disrupted what the flow had learned to exploit. Caveat: adaptation stopped at 6k by protocol; a 10k+ run was not tested, but the flat 2k→6k trend argued against it.
Second dissociation takeaway: linear decodability ≠ downstream usability — this mirrors the AR-rollout dissociation in §5. What a probe can read out is not what the consumer network actually uses.
6 · External corroboration: the field hit the same wall
UniVerse3D (CVPR'26, ShapeLLM-Omni team) — same architecture (VLM hiddens → MLP → 3D DiT), no spatial fix: reports image-to-3D REGRESSING below both TRELLIS and their own base decoder, blaming "ViT optimized for semantics not spatial resolution", with the fix left to future work. Independent evidence of the ceiling — the mechanism probe and the working fixes shown here are unclaimed there.
Qwen-Image-Edit — frozen VLM + a parallel VAE path carrying the spatial signal.
GAP3D — a VLM→DINO translation DiT inserted before the 3D generator.
Know3D — MMDiT-intermediate conditioning instead of last-layer read-once.
The field has converged on a parallel-spatial-path consensus; our position stamp is the minimal sufficient variant of it (one constant table, zero extra encoders at inference).
7 · Takeaways + next
Position is an address problem — fixable with a constant lookup table (+25% in prod).
Content-binding is an objective problem — but post-hoc internalization on a trained VLM did NOT transfer (§5b): per-patch supervision likely belongs in VLM pretraining, not in a continue-train patch.
Mimicry cannot create information — KD from a better-informed teacher is structurally unable to help.
Final deployment: dual-encoder for image mode — fusion 0.400, or add-fusion 0.380 at half the cond length. Pure-qwen ceiling for this VLM generation = 0.319 (DinoPosStamp).
Next: per-patch spatial supervision in the next VLM pretrain; quantify the VQA task-mode drift and mitigate with response-format anchoring if it persists.
single-seed, 13-14 held-out assets, IoU@64 raw; asset-level 95% CI ≈ ±0.09-0.12; orderings within ~0.03 should be read as ties — multi-seed planned for the final writeup
