Stage-1 autoregressively pretrains Qwen3.5-2B (native hybrid VLM) to predict discrete 3D tokens — a representation-shaping pretrain (VP1). The AR head is dropped at inference; a flow model generates in Stage-2. Trained bidirectionally: understanding (3D→text) + generation (image/text→3D), with multimodal replay to keep the base VLM intact (anti-forgetting).
A 3D asset is factored into three components, each a frozen VQ that quantizes a continuous VAE/SLAT latent into discrete tokens (≈3-4× compression, LLM-consumable):
3D asset ──► TRELLIS VAE ──► continuous latent ──► [densify + 3D-conv enc] ──► VQ ──► discrete tokens
COARSE STRUCTURE (SS) dense 8×16³ ──enc(↓2)──► 12³ × 1 grp ──VQ(8192)──► 1728 tok
FINE GEOMETRY (shape) sparse SLAT N×32 ──densify+enc► 8³ × 4 grp ──VQ(8192)──► 2048 tok
TEXTURE / MATERIAL(tex) sparse SLAT N×32 ──densify+enc► 8³ × 4 grp ──VQ(8192)──► 2048 tok
Total = 5824 tokens / asset
| component | role | latent in | VQ grid | tokens | codebook |
|---|---|---|---|---|---|
| SS (coarse structure) | where the voxels are | 8 × 16³ (dense) | 12³ × 1 | 1728 | 8192 |
| shape (fine geometry) | surface / shape detail | N × 32 (sparse SLAT) | 8³ × 4 | 2048 | 8192 |
| tex (texture/material) | PBR appearance | N × 32 (sparse SLAT) | 8³ × 4 | 2048 | 8192 |
1,348,757 samples. 3D tasks (understand + generate) 65%; multimodal replay (anti-forgetting) 35%. 3D data built from the ready_v2 with-material subset.
| task | input (user) | target (assistant) |
|---|---|---|
| gen_img image→3D | <image> "Reconstruct this object in 3D." | <3DSTART><3DSSSTART><SS_2786>…(1728)<3DSSEND>…<SH_…>(2048)…<TX_…>(2048)<3DEND> |
| gen_txt text→3D | "Generate a 3D asset: A low-poly hooded warrior, dark palette…" | (same 5824-token 3D sequence) |
| und 3D→caption | <3DSTART>…<SS/SH/TX codes>…<3DEND> | "A low-poly 3D character model of a hooded warrior or assassin, dark color palette…" |
| replay_text text VLM | "As a prompt generator for Midjourney, create image prompts for…" | "/imagine prompt: a logo demonstrating 'sf genius', sleek modern typography…" |
| replay_img image VQA | <image chart> "What was the total sales of beer in Alberta in 2019/20?" | "972.46" |
| check | result |
|---|---|
| VLM forgetting (VQA) | 83% = base → no catastrophic forgetting ✅ |
| 3D understanding (3D→caption) | accurate (shape/color/material/style) ✅ |
| image→3D | clean recognizable mesh (in-distribution) ✅; held-out assets still rough ⏳ |
| text→3D | simple shapes work (apple = clean red sphere); complex (rabbit/truck) still degenerate ⏳ |
Per-task loss (final ckpt): replay_img 0.15 · und 0.57 · replay_text 0.95 · gen_img/gen_txt 1.99. The ~2.77 training average is token-weighted toward the hard 3D-code prediction (5824 codes/asset over an 8192 codebook; random ≈ ln 8192 = 9.0).
