Text-to-3D feasibility — WilD3DGen

Text-only conditioning of the 3-stage flow cascade · OOD generalization · convergence · quantitative eval · 2026-07-14

text-to-3dfeasibilityeval

0 · TL;DR

Headline: text-generated 3D matches its caption almost as well as the ground-truth asset does — text→3D CLIP 0.239 vs GT-render 0.248 (96% of the caption-match ceiling), on 58 held-out assets. Its low geometric IoU (0.167) is not a quality failure — it is the expected signature of a multi-solution task ("a car", not that car).

Can the deployed 3-stage rectified-flow cascade generate 3D from text-only conditioning — no image, no DINO? Short answer: yes, the architecture supports it, it generalizes to novel single objects with correct materials, it converges by ~20k steps, and its failure modes are data/representation-bound (thin periodic topology, multi-entity composition) — not architecture-bound.

Question	Answer	Where
Does the architecture support text conditioning?	Yes — trains stably, loss reaches image-cond levels	§2
Does it generalize to unseen / hand-written prompts?	Yes for single objects (3 OOD rounds)	§3
Where does it break?	Thin / high-periodicity topology + multi-entity composition	§3
Does more training help?	No — saturates ≈20k steps (loss and visual)	§4
How good is it quantitatively?	CLIP 0.239 vs GT 0.248 (96%); deployed image baseline 0.389 IoU	§5
How to deploy text→3D at quality?	T2I bridge (inference + triplet augmentation) + native text path	§6

Held-out, real-caption text→3D vs GT. Cond is the v2.2 VLM hidden of the actual asset caption (DINO segment dropped). The generated objects are on-prompt and correctly materialed — but they are different *instances* from the GT, which is exactly why exact-geometry IoU is the wrong verdict metric for text (see §1).

1 · Setup & the question

The deployed model is a 3-stage rectified-flow cascade — SS (16³ occupancy) → shape SLAT-512 → tex SLAT-512 — normally conditioned on concat[DINOv3 spatial tokens ; connector(v2.2-VLM hidden)] via cross-attention.

Text mode = drop the DINO segment. cond = connector(v2.2 hidden of the caption); the DINO-position stamp auto-no-ops on short text sequences. No code path changes — the plain cond = connector(mask_drop(hidden)) branch handles it.

Why IoU is not the text metric. Text is inherently multi-solution — "a chair" admits many valid chairs — so occupancy IoU against the specific GT asset is a lower bound, not a verdict. The fair text metric is CLIP(render, caption) — how well the generated 3D looks like what the caption asked for — scored against GT's own render-CLIP as the ceiling. IoU is retained only for the image-conditioned baseline, where a single geometry is actually determined.

Pilot training. All three stages warm-started from the fusion image checkpoints, then trained text-only on a 4-variant caption cache (v22_tcap4, 1.69M entries: long / medium / short / long+texture, sampling weights 35/20/10/35), 2-node eff-BS-256.

2 · Architecture validity

Text conditioning trains stably and per-stage flow losses land at or below their image-conditioned warm-start levels — the flow consumes text hidden as cond exactly like it consumes image hidden.

Stage	text-only loss @plateau	image-cond reference
SS	0.168	0.15 – 0.17
shape	0.31 – 0.34	0.38 – 0.50
tex	0.45 – 0.50	0.44 – 0.51

No architectural barrier. Text hidden is a drop-in for image hidden as cross-attention cond; no new modules, no code path change, losses at image-cond levels.

3 · Generalization — the key result

Tested on hand-written prompts never seen in training, cond computed live (v2.2 forward, gen_txt template). Three rounds, increasingly out-of-distribution.

Round 1 — common single objects. Teapot, rocking chair, cactus-in-hexagonal-pot, knight helmet → clearly recognizable with correct materials. Robot dog / mushroom / camera → correct rough form. "astronaut riding a horse" → only the horse (compositional collapse to the dominant noun).

OOD round 1, common objects. Single objects are recognizable with correct materials. The compositional prompt (astronaut riding a horse, right) collapses to the dominant entity — the horse — the first sign of the multi-entity boundary.

Round 2 — far-OOD, deliberately far from the 3D-asset-library categories:

Recognizable / strong: coral reef, crumpled paper, tree stump, human heart, Gothic cathedral facade (spires + rose-window outline).
Skeleton captured: suspension bridge (deck + cables + towers), melting ice cream, Möbius strip (closed twisted band).
Fails: DNA double helix → collapses to a strand tangle (no regular periodicity).

OOD round 2, far-OOD. Compact-volume and architectural prompts (coral, cathedral, heart) generate cleanly even far from the asset library. The DNA double helix fails — a thin, regularly-periodic topology the sparse SLAT representation cannot resolve.

Round 3 — fresh set of 10. Garden hose (coiled), bonsai, desk globe on stand, pancake stack (correct count of 3) → recognizable. Sneaker detailing / treble-clef → fail (thin / flat / symbolic).

OOD round 3, fresh 10. The pancake stack gets the count right (3) — the model tracks quantity for a single object class. Thin/flat symbolic shapes (treble clef, sneaker fine detail) remain the weak class.

The boundary (reproducible across all three rounds):
Works — compact-volume objects, containers / bases, branching / wrinkled surfaces, architectural structure: anything within the training surface prior.
Fails — thin + regular + topology-defined shapes (helices, closed loops, periodic trusses) and multi-entity composition.

The thin-topology failure is the same class as image-mode's hardest asset — the rifle (IoU 0.05, §5): continuous flow + sparse SLAT under-represent thin structures. This is a representation / data-coverage property, not a training deficit or an architecture limit.

4 · Convergence — does more training help?

Loss saturates ≈20k steps. To rule out an LR artifact, after the 20k cosine annealed to ~0 we reset to a fresh cosine (peak 5e-5) and trained 18k more steps. If the plateau were LR-limited, restored LR would resume descent. It did not (200-step smoothed means, at peak LR):

Stage	pre-reset plateau	post-reset @4k (peak LR)
SS	0.169	0.168
shape	0.31 – 0.33	0.339
tex	0.45 – 0.50	0.474

No descent at restored peak LR ⇒ true objective convergence, not an LR ceiling. Caveat: flow-matching loss is velocity-MSE over random t, so a flat loss does not by itself prove visual saturation — hence the explicit render comparison below, which agrees.

Deep-training visual check at ~38k-equiv (vs 20k). No visible improvement — and the Möbius strip even degrades. Loss-verdict and visual-verdict agree: this is genuine saturation, not an under-trained model.

Geometry vs material split. SS drives geometry / topology; shape + tex drive material / surface. In progress renders, material sharpened with shape/tex steps while topology stayed capped by SS. A real bug surfaced here — the progress renders were reading a stale SS ckpt-2000; once SS actually advanced, the Möbius went open-band → closed-loop, confirming geometry does respond to SS training up to saturation.

5 · Quantitative eval (n = 58 held-out)

Set: union of the 5 held-out manifests → 60 unique, minus 2 memorized twins (≈1.000 IoU) → 58, all with real captions. Three modes:

Mode	What	SS IoU mean / med	CLIP mean
A — image-fused (deployed)	cond = [DINO ; v2.2], official sampler	0.389 / 0.382	0.246
B1 — fusion-on-text (control)	fusion ckpts fed text cond, no DINO	0.037 / 0.023	—
B2 — text-only	text-trained ckpts on text cond	0.167 / 0.072	0.239
GT (CLIP ceiling)	render of the ground-truth asset	—	0.248

Text→3D CLIP 0.239 vs GT 0.248 = 96% of the caption-match ceiling. On-prompt generation that matches the caption almost as well as the real asset does. The low SS IoU (0.167) reflects text underdetermining the exact geometry — not a quality failure.

B1 = 0.037 is the clean negative control. The fusion flows were never text-trained, so feeding them text produces garbage. This proves B2's text ability is genuinely learned, not leakage — and it is why the deployed answer trains I1+IM+T jointly (§7), rather than expecting an image-trained flow to accept text for free.

Difficulty breakdown. By GT voxel count (compact/regular vs thin/complex, median split n=29/29):

Mode	compact / regular	thin / complex
A image SS IoU	0.396	0.382
B2 text SS IoU	0.201	0.133
A / B2 / GT CLIP	0.243 / 0.240 / 0.250	0.250 / 0.238 / 0.246

Image mode is flat across geometry type (0.396 vs 0.382 — DINO anchors both). Text mode drops on thin/complex (0.201 → 0.133), consistent with the OOD boundary. But CLIP stays ≈0.24 everywhere → caption adherence is geometry-type-independent; only exact-IoU suffers.

By image-mode difficulty (easy iou_A ≥ 0.15 n=42, hard < 0.15 n=16):

Mode	easy (n=42)	hard (n=16)
A image SS IoU	0.512	0.066
B2 text SS IoU	0.214	0.044

The bimodal split is the key structural fact: on the 42 "easy" assets the deployed model is 0.512; the 16 "hard" assets (thin / blobby / featureless — rifle, crystal, wireframe) drag the aggregate to 0.389. Those hard assets still render as recognizable objects — IoU over-penalizes voxel misalignment on thin geometry (the rifle at 0.05 is a recognizable rifle). So 0.389 understates deployed quality; the honest read is "0.51 on normal objects, with a thin-geometry tail the IoU metric punishes."

58-asset in-distribution eval grid, columns [input | image-3D | text-3D | GT]. Image-3D tracks the specific asset (high IoU); text-3D produces the right kind of object matching the caption (high CLIP, lower IoU). The thin-geometry tail assets still read as their category despite low IoU.

6 · Deployment & the coverage insight

Root cause of text→3D's weakness = caption coverage. Our captions are back-labeled from existing 3D assets, so categories absent from the asset library (thin topologies, novel compositions) are simply absent from training. This is a data gap, not a model gap — and it is fixable without touching the architecture.

The T2I bridge (two uses):

text → off-the-shelf T2I (Qwen-Image / FLUX, near-unlimited coverage) → image
     → (a) inference: our fusion image→3D pipeline  → 0.40-quality 3D at T2I coverage
     → (b) training:  vanilla TRELLIS.2 image→3D    → synthetic text↔image↔3D triplets
                                                       to fill coverage & strengthen NATIVE text→3D

(a) deploys text→3D today at the deployed fusion quality (~0.40) with T2I's category coverage. (b) generates synthetic text↔image↔3D triplets to fill the caption gap and strengthen the native text path.

Prerequisite present: vanilla TRELLIS.2 is in-tree. Open risk: the synthetic-image→3D domain gap / fidelity — must be validated before scaling triplet generation.

7 · Takeaways & next

Text→3D is validated — architecturally sound (§2), generalizes to novel single objects (§3), converges by ~20k (§4), and quantitatively matches captions at 96% of the GT ceiling (§5).
The boundary is data / representation, not architecture: thin periodic topology and multi-entity composition, the same class as image-mode's rifle. Back-labeled captions are the root cause.
S3 is where the deployed text number will be measured. The deployed answer is a joint I1+IM+T model; text quality inside S3 (not this single-mode pilot) is the number that matters. B1 = 0.037 proves the joint training is necessary.
Future bets: (1) T2I-bridge triplet augmentation to fill caption coverage — validate synthetic-image→3D fidelity first; (2) AR-blueprint route (discrete 3D tokens + verifiable-reward RL) for multi-entity composition, which more flow training is unlikely to solve; (3) thin-topology representation — open whether a finer SS grid or a different structure representation helps.

single-seed; SS IoU@64 + CLIP ViT-L; text metric = CLIP(render, caption) vs GT-CLIP ceiling; 58 held-out twin-excluded. Sources: docs/TEXT_TO_3D_FEASIBILITY.md, docs/INDIST_EVAL.md (WilD3DGen / BLIP3o-NEXT).