design study — light-unify vs full Uni-VAE, staggered noise schedules · 2026-07-06
architecturedesign-studyunified-latentdiffusion
TL;DR
Verified: shape and tex latents live on IDENTICAL coords (32ch each) → one joint 64-ch flow with both VAEs frozen ("light unify") costs ~2–3 days, vs ~3 weeks for a full UniLat3D-style unified VAE.
PBR-specific risk: in cascade, texture reads the CLEAN finished shape; in naive joint denoising it reads a NOISY one → material seams may soften. Fix: staggered per-modality timesteps — δ interpolates joint(0) ↔ cascade(1) in ONE model.
1 · Three architecture options
UniLat3D (2509.25079) re-unifies TRELLIS — but on v1 + Gaussians, not PBR; its evidence doesn't transfer automatically.
A cascade = today (2 flows, texture hard-conditioned on finished shape). B light unify = ONE 64-ch joint flow on the shared coords, VAEs frozen, SS stage kept (occupancy stays editable). C full Uni-VAE = dense unified latent, single-stage — but PBR-in-one-latent is untested territory (UniLat3D only did Gaussians).
2 · Cost
16³-token flows are cheap; the 3 weeks in option C is the VAE retrain (~330k iter even with TRELLIS warm-start), where our SLAT-flow MFU optimizations don't apply.
Light unify skips the VAE entirely — ~10× cheaper than full Uni-VAE. FLOP-based at 30% MFU on 4×H200 nodes.
3 · Why naive joint may hurt PBR
The risk concentrates at crisp material-geometry seams (metal rims, wood-metal joints): texture commits while the surface is still moving. Mitigations: surface-weighted tex loss + the staggered schedule below. Note UniLat3D's plain joint DOES work for Gaussians — PBR is the open question.
4 · Staggered timesteps: δ = a cascade↔joint knob
Train with independent per-group timesteps (Diffusion-Forcing style) + dual t-embedding → one model supports every δ at inference; sweep δ∈{0, 0.5, 1} post-hoc against the existing cascade checkpoints.
t_geo = clamp(u−δ), t_tex = clamp(u): geometry denoises δ ahead, texture commits once geometry is largely resolved. δ=0 plain joint · δ=1 sequential (cascade semantics in one model, one pass).
5 · Proposed experiment & open questions
Step
What
Cost
1
64-ch joint flow, frozen VAEs, independent-t, surface-weighted tex loss (reuses the ablation-winning ① cross-attn conditioning)
~2–3 days, 4 nodes
2
δ-sweep {0, 0.5, 1} vs cascade baseline ckpts
hours
3
Judge material-geometry seam crispness + val flow-MSE + renders
—
Does PBR survive unification at all? (UniLat3D avoided it.)
