WilD3DGen · Paper study: JointDiT vs Modality Forcing
the two per-modality-timestep recipes, dissected for the unified geo-tex DiT · 2026-08-11
paper-studynext-archjointditmodality-forcing
0 · Why these two papers
They are the only two complete recipes for "one DiT, two modalities, modes dialed by per-modality timesteps" — exactly the mechanism our unified shape+tex SLAT DiT needs. JointDiT (ICCV'25) is the careful small-scale version; Modality Forcing (June 2026, FLUX.2-9B) is the scaled-up version that beats JointDiT by 57% relative on the conditional corner we care most about.
Convergent evidence. Both papers independently arrive at: freeze the pretrained stream · zero-init every new connection · per-modality timestep into each stream's modulation. Our Stage-1 design sits exactly on this consensus — plus three things neither paper has (shared voxel coords, a REAL pretrained second specialist, one-way masking).
Base: frozen Flux; RGB path + depth branch = LoRA'd copy of the SAME backbone (rank 64, α=rank/2; input stage rank 512/1024). Rank ablation is monotone: 16/32/64 → NYUv2 AbsRel 9.1/6.6/5.7.
Coupling: UniCon joint cross-attention bridges per block (bidirectional Q↔KV exchange) → zero-init linear projection → adaptive gate.
Adaptive scheduling weights (feature gate, NOT loss): w_x = σ(α(t_y/(t_x+t_y)−½)), α=3 — the noisier stream listens more to the cleaner one. Ablation: gates + unbalanced sampling BOTH needed (ImageReward Rank1 30.7% vs 21.2% weights-only).
Timestep sampling: exact 50/50 — half diagonal (one shared t~f), half independent f×g; shifted-sigmoid transform with s=3.1582 (kohya Flux default) and s=0.25. NO explicit corner sampling (grep-verified) — conditional modes are emergent.
Inference: 3 modes purely by timestep dialing, 20 steps, no ensembling; 10% text-drop CFG for the text condition only.
Costs: +19.8% params, 1.6s→4.7s sampling; depth still trails specialist estimators.
Cross-stream timestep mixing: zero-init embedders let each stream's modulation see the OTHER modality's timestep.
Self-distillation anti-drift: pass noisy RGB through the frozen T2I teacher, penalize velocity deviation with weight λ_hi·t_depth + λ_lo·(1−t_depth) — strongest when depth is pure noise (inputs the teacher knows), relaxed when depth context appears.
Sparse data trick: missing depth pixels filled with Gaussian noise = "no data here" signal → trains on 17M frames / 58k scenes of REAL sparse-depth video (TartanAir v2, ARKitScenes, Waymo, …).
Ordering ablation: trajectory warp f_α(t)=αt/(1+(α−1)t), α∈[2⁻⁵,2⁵] — denoising RGB (leader) first gives more consistent depth; "RGB acts as a scratch pad".
Scaling: 370M→3.3B from-scratch DiTs, 0→1.92B images: bigger T2I + more data = better depth, monotone.
Two coupling philosophies: JointDiT = per-layer feature bridges (tight adherence, +19.8% params); Modality Forcing = append a stream, attention-only coupling (scales, but looser conditioning by its own admission). Both freeze the pretrained stream and zero-init every new pathway.
3 · The decisive disagreement: corner sampling
JointDiT: smooth f/g densities only — the conditional corners are never trained exactly, and its depth estimation lands at AbsRel 5.12 (NYUv2).
Modality Forcing: 20%+20% of training batches pin the conditioning modality at EXACTLY t=0 — and reaches 2.52, a 57% relative improvement, matching the MoGe-2 specialist (2.89).
Scale differs too (9B vs 12B-LoRA, 17M frames) so not a pure ablation — but the corner-mass recipe is the single largest methodological difference, and MF's authors motivate it exactly as corner coverage.
For us: our tex|mesh product mode lives at the t_shape=0 corner → corner mass is DEFAULT-ON in Stage 1 (we run 40% vs MF's 20% as ablation A1).
The training coverage difference in one look: JointDiT (left) leaves the edges to chance; Modality Forcing (right) pins 40% of all batches onto the two conditional edges. Our product modes live on those edges.Left: conditional-corner quality (depth from image) — explicit corner training + scale closes the gap to the dedicated specialist. Right: the reverse corner — MF wins FID but admits looser depth adherence than JointDiT's feature bridges; adherence is exactly what our concat_cond pathway is for.
real pretrained tex specialist — strictly better start
Corner sampling
none
20% + 20%
40% t_s=0 (A1: vs 20%)
Timestep density
shifted-sigmoid s=3.1582 / 0.25
logit-normal μ=1.1 (+plateau for depth)
uniform per branch (matches our flows' pretraining)
Cross-t visibility
—
zero-init mixing embedders
t_s into tex AdaLN (zero-init)
Anti-drift
backbone frozen
self-distillation vs frozen teacher, λ(t) schedule
Stage 2: self-distill vs frozen fusion_shape/tex
Ordering
—
f_α warp; leader-first wins
α-warp lag schedule (shape leads) — direct precedent
Missing data
—
noise-fill = "no data" signal
Stage 2: 455k tex-less assets via noise-filled tex
Modality alignment
pixel-aligned RGB/depth
not even same token space
1:1 shared sparse voxel coords — strongest pairing of the three
5 · What we take, what we don't
Take from MF: explicit corner sampling (default-on), α-warp leader-first scheduling, self-distillation for Stage-2 unfreezing, noise-fill for tex-less assets, cross-timestep visibility.
Take from JointDiT: zero-init discipline on every new pathway, the adaptive gate idea (if A2 feature bridges are added), the convention-trap warning (both papers are t=0-noise; TRELLIS is t=0-clean — re-derive, never sign-flip).
Take from neither (ours): one-way hard mask (shape never sees tex — enables full freeze + no marginal training), concat_cond hard geometric coupling (tighter than either paper's mechanism — MF's own D2I looseness is the cautionary tale), real-specialist warm start for the second stream.
Don't take: MF's bidirectional attention (would unfreeze shape + require marginal corners for mesh-only); JointDiT's LoRA-everything (our tex stream deserves full flow_tune-last20, it is a real specialist); MF's pixel-space second modality (irrelevant — our modalities already share a latent family).
Net effect on the plan: Stage 1 unchanged in structure, upgraded in defaults — corner mass on, α-warp scheduler, and Stage 2 gains a concrete anti-drift mechanism. The two papers between them de-risk every timestep-side decision; the remaining genuinely-open items are architectural (one-way mask, concat_cond vs feature bridges — our A2/A3 ablations).
6 · The exact architecture we will build (walkthrough)
Start = our two existing SLAT flows (column ①: a plain DiT — one modality, one timestep, self-attn + cond X-attn).
Merge them MF-style (column ③): put shape tokens and tex tokens into ONE joint attention per block; each modality keeps its OWN weights (from fusion_shape / fusion_tex).
3 surgical changes, nothing else: (a) one-way attention mask — shape queries never see tex keys (✗ cell), tex→shape cross logits gated by λ init −10; (b) tex's concat_cond now fed the shape stream's CURRENT state x_{t_s} instead of clean GT; (c) zero-init t_s term into tex's AdaLN.
Everything else is untouched: shape stream fully frozen (≡ specialist forever), tex self-attn/MLP/cond X-attn weights warm-started, 3D RoPE shared per voxel pair.
Left→right: what changes at the block level. The attention-mask grids are the whole story: standard DiT is trivial; MF opens everything bidirectionally; ours keeps one cell BLOCKED (shape never reads tex) and one cell GATED (λ, zero-init) — that single ✗ is what buys a fully frozen shape stream and risk-free mesh-only mode.
sources: arXiv 2505.00482 (+ICCV supplement, verified via 103-agent deep research) · arXiv 2606.13676 · companion design report: unified_geotex_dit.html · 2026-08-11
