MolmoAct2 → BLIP3o-NEXT: VLM-side 3D Pretraining + Per-layer KV
Design / hypothesis · adapting MolmoAct2 to the TRELLIS.2 + Qwen cascade · 2026-06-20
designhypothesisvlm3d-pretraineditingper-layer-kv
TL;DR
DESIGN / HYPOTHESIS — not validated. "VLM-based 3D pretraining helps gen" has no isolating ablation (MolmoAct2 ships it, doesn't prove it). Test the cheap piece first.
Mechanism: replace read-once cross-attn with per-layer KV (MolmoAct2-style) — fixes the multi-view / text→3D bottleneck.
Two new VLM stages:VP1 3D-understanding + VP2 editing pretrain → a 3D-literate Qwen.
Pays double: one discrete-3D tokenizer = 3D-literate KV for gen + native 3D input for editing.
1 · Read-once cross-attn → per-layer KV
The core change. Discrete path is training-only; at inference the flow expert generates from the KV.
Left: today's connector(final-hidden) is read once by every TRELLIS block → static K/V → multi-view blur. Right: each flow block ℓ cross-attends Qwen layer ℓ's KV via P_K/P_V adapters, detached (frozen Qwen ⇒ insulation for free).
2 · Pipeline — two new VLM-side stages
VP1/VP2 train the VLM with cheap next-token loss (not flow-loss-into-Qwen); the cascade stays.
VP1 makes Qwen 3D-literate (predict discrete 3D tokens). VP2 feeds an input asset's 3D tokens for editing (the Plan-2 bridge). Flow cascade conditions on Qwen-3D via per-layer KV.
3 · 3D tokenizer — the Kyvo-style route (verified vs Kyvo's code)
A 2nd-stage tokenizer on the frozen TRELLIS.2 VAE: densify sparse SLAT → 3D conv → VQ → ~512 tokens/modality. Tokens are training-only (dropped at inference), so lossy is fine. FAST does NOT transfer (action/temporal-specific).
TRELLIS.2-512 SLAT (sparse 32³×32) → densify+occ-mask → 3D conv U-Net (32³→8³) → VQ-8192 → 512 tokens; recon target = frozen-VAE decode of z. Shape & tex get SEPARATE tokenizers; SS = occupancy (skip). 512 is fine for VP1/geometry; texture is higher-entropy → likely needs a bigger budget. KEY: Kyvo's released code is a 2D-image VQGAN with NO texture VQ — we build the 3D tokenizer by adapting taming-transformers 2D→3D and make texture ourselves.
4 · Validation order (cheap-first)
Let each piece earn its place; honor V3's "pure flow already works".
Step 1 (per-layer KV, frozen Qwen) may already fix it → no VP1 needed. Only escalate to VP1 → richer VQ → VP2 if it plateaus. Avoid flow-loss into Qwen (D2).
