The backbone is TRELLIS.2-4B: a cascade 3D generator
(SS sparse-structure flow → shape SLAT@512 → texture SLAT@512), natively conditioned on
DINOv3 visual features through a cross-attention DiT (the 3D latent is the main
sequence; the condition is read as K/V by cross-attention).
Goal: replace the condition with representations from Qwen3.5-2B (a VLM), to drive
3D generation from a single unified interface — {single-image, multi-image, text} → 3D —
and lay the groundwork for 3D editing / conversational generation.
2 · Model Architecture
The cascade backbone. The condition enters every DiT block once via cross-attention (as K/V).
Key property: the condition is a read-only dictionary — it does not co-evolve with the
3D latent across layers (contrast with MMDiT/joint-attention, §6).
How we evolved the conditioning. (1) native DINOv3; (2) Qwen-only via a small connector (V3);
(3) the current best: concatenate raw DINOv3 tokens with connector(Qwen) into one cross-attn,
plus DINO-dropout and (for multi-image) a per-view embedding.
3 · Exploration Timeline
V3 · Condition swap Qwen last-layer hidden → small connector → replace
DINOv3. Works, but geometric alignment is limited (Qwen's last layer is semantic, weak on geometry).
Infra · VLM cache + split training Precompute the frozen VLM hiddens to
disk (no VLM at train time); train the three cascade components separately. ≈5× throughput.
Fusion · DINO × Qwen in one cross-attn
cond = [raw DINOv3 tokens ; connector(Qwen)] concatenated into the same cross-attn, with a
DINO-dropout curriculum. Beats Qwen-only across the board; reaches teacher-level fidelity.
Multi-task · I1 + IM + T One set of weights handles three input types
(single-image / multi-image / text). I1 fidelity preserved; IM works; T (text→3D) path alive but weak.
Multi-view deep dive Diagnosed IM "mushiness": ruled out resolution;
isolated narrow training view-coverage + OOD viewpoints; in-distribution multi-view is in fact effective.
4 · Key Results
4.1 Fusion reaches teacher-level fidelity
Result: feeding raw DINOv3 tokens and Qwen together into the same
cross-attn (DINO bypasses the connector, keeping its native distribution) beats Qwen-only at
half the steps — the dome becomes solid, the L-shaped lot reappears, texture hallucinations vanish.
Stable at both CFG settings.
Fusion @3000 final. Columns: input | TEACHER (DINOv3) | Qwen-only (S2) | Fusion | Fusion (CFG5) |
Qwen-only ablation. Fusion matches TEACHER fidelity; the Qwen-only column shows the dome opening a
hole, camo texture on the teapot, etc.
4.2 Ablation: who carries the quality?
On single-image: Fusion ≈ DINO-only (Qwen's marginal ≈ 0), while
Qwen-only ≈ the Qwen baseline (the path stays alive, not damaged by fusion training). So single-image
fidelity is carried by DINO; Qwen is a "harmless passenger" on single-image and earns its keep on
multi-image / text tasks.
Segment ablation. Columns: input | Fusion (both segments) | DINO-only | Qwen-only | TEACHER.
Fusion ≈ DINO-only on single-image; Qwen-only still produces an object (the path is alive).
4.3 Multi-task: one model, three input types
Stage-3, three tasks. Top: I1 (single-image, fidelity preserved) / middle: IM (multi-image) /
bottom: T (text → 3D). T produces coherent 3D from text alone (non-trivial), but text-to-geometry
alignment is still weak (frozen Qwen + only last-20 tuned + text weight 0.2).
5 · Multi-view (IM) — see the dedicated report
The multi-image / multi-view investigation (why IM looked "mushy", and what actually causes it)
is written up separately so it can stand on its own: "Multi-view (IM) Diagnosis"
(on the hub dashboard). In short: it is not resolution, and not "multi-view is broken" —
the bottleneck is training view-coverage / OOD robustness (a data issue). In-distribution, multi-view
helps.
6 · Where this sits in the literature
Three "frozen-VLM → generative model" works point to the same signal:
Work
How the condition is injected
MolmoAct2
Per-layer KV-cache: the expert reads the VLM's K/V at the same depth, every layer
EVA-01
Concatenated sequence + global self-attention (3D tokens see vision tokens at every layer)
Last-layer hidden → small connector → single cross-attn (condition is a read-only dictionary)
Common thread: the ones that work let the condition tokens participate in per-layer
joint attention, rather than being queried once by cross-attention. This is the clear direction for
our next architecture upgrade (cost: TRELLIS is a cross-attn DiT; moving to joint-attention is major surgery).
7 · Conclusions & Next Steps
Fusion (DINO × Qwen) is the current best recipe: teacher-level single-image fidelity +
a live Qwen pathway + a unified multi-task interface.
End-to-end pure Qwen: marginal on single-image; its value must come from T / IM / editing,
which still need investment.
IM is not "broken": effective in-distribution; the bottleneck is view coverage
(raise max_views) + OOD robustness.
T is weak: limited by a frozen Qwen + a small training budget; next is to raise its weight /
unfreeze later Qwen layers.
Long term: evolve toward per-layer joint attention (MMDiT / double-stream) so the condition
truly participates in generation.
molmo3-3d · auto-generated exploration summary (images are real evaluation renders,
single seed, pinned views). Infra: VLM-hidden cache + 3-component split training (~5× speedup).
