read-once cross-attn vs MMDiT co-evolving vs per-layer KV · VP1 v2.2 taps · 2026-07-05
ablationMMDiTconditioningshape-flow
TL;DR
At the 1.3B / 45k-asset budget, the single-stream read-once cross-attn baseline (①) is the best conditioning mechanism (val 0.302). Adding conditioning-path complexity — per-layer adapters (③) or a co-evolving dual-stream MMDiT (②) — does not help; MMDiT is marginally worst.
The MMDiT deficit is not an initialization artifact: fairly inheriting TRELLIS's condition-reading (add_k/v ← cross_attn.to_kv) moves val by 0.0001 (0.32692 → 0.32680). It survives fresh-vs-inherited, longer training, and holds on held-out, in-distribution, and truly-novel (SketchfabV1) data.
Consistent with DiT-Air (Apple, 2025): MMDiT's dual-stream QKVO/MLP duplication is largely redundant at a constrained budget; a parameter-efficient single stream matches or beats it.
1 · Setup
Testbed = the 512-resolution shape SLAT flow (sparse DiT, 1.29B). Conditioning = per-token hidden states ("taps") from the frozen VP1 v2.2 3D-VLM (Qwen3.5-2B + 3D code tokens) run on [image + teacher-forced GT codes]. All arms: eff BS 256, AdamW 1e-4, 8000 steps, 10% CFG cond-drop, fixed GT structure coords (isolates the SLAT-feature conditioning; the SS structure is held constant). Every arm starts from the pretrained TRELLIS weights.
① cross-attn: the stock TRELLIS block, read-once cross-attention. Condition path fully pretrained.
③ per-layer KV: block machinery kept verbatim; each block's cross-attn reads the VLM's depth-aligned tap through a small adapter (+63M). Read-only, fully-inherited condition-reading.
② MMDiT: SD3/Qwen-Image double-stream joint attention; the condition co-evolves through its own QKV/FFN. Verified equal to diffusers JointAttnProcessor2_0 at the math level; zero-init gate ⇒ bit-exact pretrained backbone at step 0.
① is best by ~8%. ③ and ② cluster ~0.32; the ②-vs-③ gap (0.003) is within single-seed noise and not claimed.
Held-out val flow-MSE. ① (baseline) clearly lowest; ②/③ plateau ~0.327/0.324. Fresh and inherited MMDiT are indistinguishable.
3 · Rendered quality — held-out complex objects
Fixed GT structure → decoded mesh → normal-map render. Differences are subtle (structure is given); ② is marginally noisier on fine surface detail (barrel flower, machine-box interior, moon-rover).
input | GT | ① cross | ② MMDiT | ③ per-layer KV — 6 held-out objects. All produce coherent shapes; ② noisiest on detail.
4 · Novel data (SketchfabV1) — TRELLIS likely never saw these
To rule out pretrained-decoder memorization, we tokenized + tap-extracted a fresh batch from the team's own SketchfabV1 set (not in any training manifest). Ranking holds; ② degrades more on novel data (laptop screen, bust hair, container).
SketchfabV1 novel objects. ① cleanest; ② noisiest — the ranking generalizes to unseen data.
5 · In-distribution (train-seen) — rules out clean overfit
On training-seen objects, ② is also noisier (crate, dark box) and its train-loss (0.293) exceeds ①'s (0.275). So ②'s deficit is not clean overfitting (which would be clean-on-train, noisy-on-held-out) — it fits training worse too.
Training-seen objects. ② noisy even here → low training efficiency, not memorization.
6 · Fairness correction — the key control
An early MMDiT built its condition path from scratch (a design regression — the whole point of ② was to inherit TRELLIS). We rebuilt it so the c-stream K/V projections directly inherit the pretrained cross_attn.to_kv (c-stream width = cond_channels 1024, matching TRELLIS dims; structure stays Qwen-Image). Verified: gate=0 ⇒ bit-exact pretrained; add_k/v == cross_attn.to_kv split.
Result: fair inheritance changed val by 0.0001 (0.32692 → 0.32680). The deficit is architectural, not an init artifact. A deeper tension surfaced: perfect inheritance ⟂ co-evolution — pretrained cross-attn reads a fixed raw cond; MMDiT's c-stream reads an evolving one, so a co-evolving stream can never perfectly reuse read-only pretrained weights. ③ can (it stays read-only); ② inherently cannot.
Fair final: GT | ① cross | ② MMDiT-inherited-8k | ③ per-layer KV. Inherited ② still marginally noisier — architectural, not init.
7 · Pipeline sanity (overfit certification)
Before the full runs, each new arm passed a cond-dependent-target overfit test: loss must beat the cond-blind floor, provable only through the conditioning pathway. Both ② and ③ passed (loss → 0 past the floor; zero-init gates self-open).
Overfit certification — cond pathway is trainable; zero-init gates open by gradient.
8 · Takeaways
Robust: at 1.3B / 45k, single-stream read-once cross-attn (①) is the best conditioning mechanism; extra conditioning-path parameters (③ or ②) don't pay off. Aligns with DiT-Air.
Robust: MMDiT's deficit is architectural — fresh vs inherited, longer training, and held-out/in-dist/novel data all agree (~0.327). Not an init or overfit artifact.
Nuance: in absolute quality all three produce clean, usable shapes; the 8% val gap reads as subtle surface-detail noise under fixed-structure evaluation.
Open: ②-vs-③ ordering (0.003) needs multiple seeds; MMDiT's co-evolution is known-good at large scale (SD3) — untested here. Data-scaling to ~350k SketchfabV1 is the next lever to test whether more data lets the co-evolving stream reach its potential.
