WilD3DGen · Unified Geo-Tex DiT — the next architecture

0 · TL;DR

1 · The gap — why this is worth doing

• Unified-latent camp (UniLat3D, 3DTopia-XL, TRELLIS v1): one DiT, but geo+tex are fused per-channel in a single latent → cannot hold geometry fixed while regenerating texture.
• Cascade camp (TRELLIS.2 = our base, Hunyuan3D 2.x): mesh-only / tex-only exist, but as two separate 1.3B models with GT-shape teacher-forcing (exposure bias) and inference-only coupling.
• Our shape/tex SLATs live on the same sparse voxel coords — a 1:1 per-voxel pairing neither camp has. Unify at the model level, keep the latents separate.

2 · Architecture

• SS flow untouched (defines the coords). The two SLAT specialists become two streams of one DiT: each keeps its own weights + AdaLN fed by its own timestep.
• Streams talk via UniCon-style joint cross-attention (bidirectional KV exchange) → zero-initialized projection → adaptive gate w(t).
• Warm start: shape stream ← fusion_shape ckpt, tex stream ← fusion_tex ckpt. Zero-init bridge ⇒ at step 0 the joint model exactly reproduces both production flows — training can only grow coupling, not break the specialists.
• Cond path ([DINOv3 ; VLM hidden] read-once cross-attn) is unchanged; all caches and the S3 tri-modal data mixture are reused as-is.
• Bonus: assets without tex (455k total vs 293k with tex) train the shape stream via modality masking — data utilization goes UP.

3 · The mode dial — one model, five modes

• ① Joint (t_s = t_x): one pass produces geo+tex; tex sees the shape stream mid-denoise → the cascade's exposure bias disappears.
• ② Tex | mesh (t_s ≡ 0): shape tokens are clean encoder outputs, never integrated → geometry preservation is exact by construction. This natively covers E1a/E1b re-texturing and bypasses the SLAT-renorm locality trap.
• ③ Geo | tex (t_x ≡ 0): free reverse capability.
• ④ Mesh-only (t_x ≡ 1): tex branch FED pure noise (tokens present, not dropped) — UniDiffuser's marginal theorem; exact under rectified flow.
• ⑤ Tex-only (t_s ≡ 1): rarely useful alone, comes free.

4 · Verified training recipe (what we implement)

5 · Corrections vs our first manual read + honest costs

• Corrected: we initially planned explicit 4-corner sampling "per JointDiT" — verified: JointDiT uses none. But its conditional modes are side capabilities; ours are products → corner oversampling is our ablation #1, not an assumption.
• Corrected: the convention flip is not a sign flip — the gate denominator changes t_x+t_y → 2−t_x−t_y. Unit-test (x_t, v-target) per branch before anything else.
• Corrected: JointDiT warm-starts ONE frozen specialist + LoRA copy; our two-independent-checkpoint fusion is genuinely untested territory (the zero-init principle transfers; the rank numbers do not).
• Honest cost (verified): JointDiT pays +19.8% params, 1.6s→4.7s sampling, and its joint depth still trails specialist estimators on ETH3D — gate experiments must show ≈ parity with the current fusion flows, not assume it.

6 · Gate experiments (round 1)

• G0 sanity: zero-init assembled model at step 0 ≡ both fusion flows (bit-exact forward).
• G1 joint: ≥ cascade baseline on the 14 clean held-out assets (same SS, same cond, same seeds).
• G2 tex|mesh: material-L1 ≈ current tex flow; geometry bit-exactness checked at the decoder.
• G3 mesh-only: geom metrics ≈ current shape flow.
• A1 ablation: + explicit corner oversampling (10–20% t_other pinned) vs pure f/g — the open question the papers cannot answer.

Full adversarially-verified deep-research dossier (all claims, votes, verbatim evidence, 21 sources): deepresearch_full.json · also archived at BLIP3o/docs/UNIFIED_GEOTEX_DIT_deepresearch.json.

7 · Dual-mode implementation plan — both cond modes in one codebase

• One flag: cond_mode ∈ {cross_attn (A), stream (B)} in unified_geotex.py — the Stage-2 A/B is a config change, not a rewrite.
• A (default, Stage 1): cond stays read-once cross-attn (the June-ablation winner). New params ≈ +5M (+0.2%): λ gates (360 scalars) + zero-init t_s→tex mixer. Tex-attn keys ×2; shape stream untouched.
• B (Stage-2 arm): cond promoted to a third stream — zero-init proj 1024→1536 + 30 cloned blocks (self-attn 9.45M + MLP 25.18M each) ≈ +1.06B trainable, no pretrained source. Cross-attn KEPT during transition (warm-start safety); pruned only if B wins. Default gate: cond never reads voxels → cond stream is sample-constant → per-block K/V cacheable across all ODE steps.
• Params (measured from ckpt tensor shapes): flow = 1292M (MLP 755 · self-attn 283 · cross-attn 236 · shared adaLN 14). A: 2.59B total / 0.87B trainable. B: 3.65B total / 1.93B trainable.
• Files touched: trellis2_blip3o/unified_geotex.py (new, ~400 LoC: UnifiedBlock + group-gate matrix + CondStream) · flow_heads.py (+unified loss entry) · dataset_native.py (+timestep-pair sampler, noised concat_cond) · train_native.py (+model_type branch) · scripts/ (G0 test, 3-mode sampler, G1/G2 eval).
• Evidence bar for B: it re-fights the June cond-injection ablation (read-once won, MMDiT-style worst, fairness-controlled) — B ships only if it beats A same-data-same-steps.

8 · Final decision log (2026-08-11, design frozen)

Engineering source of truth: BLIP3o/docs/UNIFIED_GEOTEX_DIT_DESIGN.md (full recipe, gates G0-G2, ablations A1-A5, footguns).

9 · Training config & time scheduling (finalized)

10 · AS-BUILT (2026-08-11) — implementation status, supersedes conflicting details above

• Coupling (geo→tex): coupling="union" DEFAULT — MF-verbatim bare single-softmax over [K_x;K_s] (dit.py:137-148). coupling="gated" READY FALLBACK — in-house mmdit_slat.py:190-196 two-softmax + per-head zero-init gate. Decision is empirical: G0 measures the per-block cross/self logit ratio on the real checkpoints (mmdit_slat measured O(1e2-1e3) for its independently-trained pair; MF's clone streams are naturally matched) — O(1) clears union, pathological switches to gated before any training run is spent.
• Sampler: 20% exact t_s=0 corner (MF-exact per-corner mass) + 80% independent uniform² — MF-adapted; the dropped d2i corner's budget folds into joint (that mode doesn't exist here). 40% corner and the α-band are ablations (A1/A6), not defaults.
• Deleted after audits: zero-init bridge Linear (+71M), trained λ logit bias, runtime gates w(t), bridge-dropout, renorm adapter (the two norm-stat sets measured bit-identical — guarded by a unit test), tail-only training.
• Blind-audit fixes: cond key-masks now consumed (drops were silent no-ops), autocast wrappers (bf16 TimestepEmbedder crash), EMA overlay on checkpoint loading (published-number parity), use_checkpoint honored, coords-parity asserts, t/target dtype discipline, alpha_lo bug, train() keeps geo eval, RoPE-parity asserts, stream-axis decision documented (we match JointDiT; MF's 4th RoPE axis is warm-start-incompatible with a pretrained 3-axis head_dim split).
• Verification: CPU unit tests 6/6 · imports clean · G0 (GPU, queued) = dual-coupling identity + the [SCALE] measurement → 100-step sanity → 3k Stage-1 → G1/G2.

11 · BIDIRECTIONAL PIVOT (2026-08-12) — final S1 topology, supersedes the one-way verdict of §4/§10

• Corner-masked bidirectionality (Diffusion-Forcing noise-conditioned-mask precedent): per-sample, geo's tex-read is HARD-disabled at the t_s=0 corner — exactly the tex|mesh operating point — so geo stays time-invariant there: the 30-block K/V one-pass cache and the exact geometry lock survive bidirectionality. In the joint interior (t_s>0) both streams read each other, standard-MMDiT phase order (both QKV from pre-update hiddens, then both advance).
• Dual corners (full MF recipe restored): 40% t_s=0 EXACT (flagship tex|mesh; user doubled MF's 20%) + 20% t_x=1 EXACT (mesh-only MARGINAL — tex lane carries pure noise; diffuse at t=1 returns ε exactly, no leakage) + 40% independent uniform square. Hard unit-asserted: 4×4 occupancy of the joint square; both corners span the opposite axis.
• Stream tag: TRELLIS RoPE pads 63 coord-driven pairs (21/axis) to 64 with polar(1,0) — an IDENTITY pair that is the 128/2/3 integer-division remainder (rope.py L47-51, code-verified). Foreign keys get that pair rotated by π/2 at the borrow site: float-EXACT (swap+negate), within-stream attention bit-identical, cross-stream logits gain an antisymmetric which-stream phase. Adapted from MF's stream-id RoPE axis (a literal 4th axis would re-split the pretrained head_dim = warm-start killer). Fallback if 2/128 dims prove weak: zero-init additive segment embedding.
• Symmetric cross-t mixers: tex has the MF-v2 mixer (embed(t_x) + α·mixer(t_s)) since v1; geo now gets the mirror (embed(t_s) + α_s·mixer_s(t_x)). Both gates zeros(1) (MF FSDP note), both mixers FRESH TimestepEmbedder instances (MF dit.py:277-288), mixed at the vec level once before the (frozen) adaLN. The gradient asymmetry is the LOSS's, not the architecture's: α has a direct path; α_s reaches the tex loss only through the cross-stream read (in-graph frozen geo lane). Watch-trio at ckpt-1000: α, α_s, b_gates opening.
• S1 training unchanged in spirit: geo WEIGHTS frozen, loss = tex velocity only; the geo lane runs in-graph so the new zero-init parameters receive gradients. Whole block pair gradient-checkpointed (~92GB/GPU measured, below the one-way run's 137GB).
• G0 re-certified after the pivot: 6 arms × (L1-tiny + L2-real 1.3B) all pass. One real catch fixed on the way: union-bidir corner rows must take the FLASH pure-self result — per-sample sdpa is 1-ULP-class off flash at real scale (measured 4.7e-2 over 30 blocks); corner rows are now kernel-identical to the cached inference path.
• mesh-only in the bidirectional world: two implementations ship — (a) the pure geo specialist path (always available; geo weights untouched in S1), (b) the marginal (tex pinned at t_x=1 with fresh noise per step, the trained corner2 config), primary once S2 trains geo.

12 · S1 RESULTS (2026-08-12) — trained, gated, judged

• Run: runs/geotex_s1_v1, 3000 steps in 7h08m (avg 8.56 s/it), loss 0.7524 → 0.5295 (min 0.4808). Config: corner-masked bidirectional + fused MMDiT attention + absolute RoPE-pad stream tag + symmetric cross-t mixers + dual corners 0.4/0.2 + union coupling + FULL 30-block tex training (1301M trainable) + elastic GC + bf16 with fp32 grad accumulation + ZeRO-1, bs4×ga8.
• The two zero-init gates both opened, monotonically: cross_alpha (tex knows t_s) 0.0081 / 0.0153 / 0.0190 at ckpt 1k/2k/3k; cross_alpha_s (geo knows t_x) 0.0114 / 0.0206 / 0.0231. The geo-side gate — whose gradient must cross the stream boundary through frozen weights — grew FASTER, so the starvation worry was unfounded. The bidirectional timestep awareness is genuinely used, not decorative.
• Multi-seed: 3 seeds stay stable and diverse, and the warm-start union perturbation that was plainly visible at 3 steps (washed-out / green / dark textures) is fully absorbed by 3000 steps — the MF-faithful "stitch bare, let training absorb it" bet paid off at our scale too.
• G1 hypotheses (ranked, to be diagnosed not guessed): (1) concat_cond distribution shift — joint feeds tex the NOISY geo state x_ts while 40% of training sits at the clean t_s=0 corner, so raising the corner from MF's 0.2 to 0.4 may have starved joint-region coverage (A1 is the ready control arm); (2) the α-warped grid makes geo's early steps too coarse; (3) joint training covers the whole square, so any single α curve is thinly sampled; (4) geo reading high-noise tex pollutes it.
• Methodology note worth keeping: every verdict here is a number AND my own image read. Latent MSE alone previously ranked a visually worse model higher (n=2), and G1's failure is invisible in the summary statistics — only the renders show the collapse.

13 · TIMESTEP SAMPLING CORRECTED (2026-08-13) — the interior 40% was aimed where inference never goes

• How the deviation happened (on record): the design doc §5 specified 40% t_s∼U[0,t_x] as a lag regime. When the bidirectional topology added the second corner (20% at t_x=1), the lag arm was displaced by an independent-square draw and the doc was never updated. Caught only because the owner asked whether the square wastes capacity — it does.
• What the references actually do (re-verified, not recalled): MF = 60% joint with BOTH timesteps drawn independently (logit-normal μ=1.1) + 20% I2D + 20% D2I; JointDiT = 50% diagonal + 50% independent, no corners; UniDiffuser = 100% independent uniform. None restricts the joint region to a triangle, so the square was defensible — but none of them has our asymmetry either: all three of our modes are one-sided in the leader.
• A claim retracted: an earlier verbal justification held that MF’s α=32 inference “works off the full-square coverage”. The papers say no such thing; that was an inference presented as provenance. The verified facts are the mass fractions above, nothing more.
• Why uniform-over-triangle and not the α-band: the band arm draws α log-uniform and places t_s on the exact f_α^-1 curve. For large α, t_s ∝ 1/α, so log-uniform α gives density ∝ 1/t_s — mass piles up next to the t_s=0 corner that already holds 40%. Uniform over the triangle covers every α the inference sweep may pick without double-investing where we are already saturated.
• Shipped as: p_lag=0.35, p_marg_s=0.05 in flow_heads.sample_timestep_pairs, both defaulting to 0 — the defaults were regression-tested bit-exact against the pre-change sampler, so no earlier run is retroactively altered. Measured over 200k draws: t_geo=0 40.1% / t_tex=1 19.9% / upper-triangle interior 35.0% / tex-ahead 5.0% (all of it the t_geo=1 edge).
• Cost of switching mid-run: S2b restarted at ~1.2 h in. The three S2b gates survive the change because none of them is an S1-vs-S2b comparison — G3 is against the shape specialist, G2 against the tex specialist, G1′ is joint-vs-mesh-only inside the same model. What the restart forfeits is the narrower question of how much the geo unfreeze contributed on its own.

14 · UNIFY DiT — FINAL ARCH / CONFIG (2026-08-15)

Config

The four deliberate deviations

• Segment tag on the spare rope pair, not on a rope axis of its own — MF gives the modality a 4th axis; measured, that would take each spatial axis from 21 pairs to 16. TRELLIS already leaves one pair unrotated, so the tag is free, and the pair's contribution to a cross-segment logit is cos(90°×Δid): adjacent ids null it, ids two apart negate it — either way a decisive mark, versus the ~1 rad a frequency ladder gives on its fastest pairs.
• Ratio 1:2, not MF’s 1:3 — owner decision; Hunyuan3D-2.1 also uses 1:2 (16 double + 32 single), so both ratios have precedent.
• Four segment ids, including DINO vs Qwen — MF gives txt the SAME time_id as img (0) and separates it by using a different RoPE axis (`_txt_ids` puts arange(seq) on axis 3, images put h,w on axes 1-2). We have three spatial axes in use and no free one, so separation has to happen on the segment axis itself. The DINO/Qwen split is free capacity: they are two different encoders concatenated into one sequence and nothing else told the model which was which.
• cond carries no sequence position — MF’s txt does. Ours would have to borrow a spatial axis, colliding cond index 5 with voxel x=5. The bet is that ordering is already baked in by Qwen’s own RoPE and DINOv3’s position embeddings. One line to flip; deferred to an A/B.
• cond modulation = sum of BOTH diffused streams’ timestep embeddings — all three references feed the condition stream the (single) diffused stream’s timestep through its own projection; we have two diffused streams, and summing keeps cond informed in every mode (tex|mesh moves t_x, mesh-only moves t_s) instead of picking one arbitrarily. Two SEPARATE embedders, so the sum is asymmetric — measured at init, the swapped pair (t_s,t_x)↔(t_x,t_s) sits 5× farther apart than the nearest legitimate neighbouring pair.

What the sparse latents force that FLUX / Hunyuan never had to solve

• Varlen everything. Hunyuan3D’s latent is a fixed-length unordered vecset, which is why it can run dense sdpa with pe = None. Ours is SLAT: sparse voxels, 432–8013 per asset (median 2222), carrying real coords. The joint sequence is built per sample and fed to TRELLIS’s own varlen flash — the same kernel every SLAT block already uses. “Sparse” here means the TOKENS are the active voxels; attention among them is attn_mode="full".
• Per-modality timestep in a shared stack. FLUX/Hunyuan broadcast one vec; our whole mode matrix (mesh-only / tex|mesh / joint) exists only because geo and tex carry independent timesteps. Shared blocks therefore share WEIGHTS but not MODULATION — which is exactly what MF’s _stack_per_token_mod does after its merge.
• Two generated streams on identical coordinates. MF’s depth is spatially aligned with RGB and it needs a modality axis for that reason; ours is the stronger version of the same problem — geo and tex are not merely aligned, they are on the SAME voxels.
• Sequence length. Joint sequence = 2×2222 + 2074 ≈ 6518 tokens at the median, 26,624 worst case. cond is 32% of it — FLUX/Hunyuan text is 77–512 tokens. Merging saves PARAMETERS, not compute: attention is identical in both regions, only the weights differ.

Bugs the audit caught (all fixed and re-tested)

• 🔴 No cond key mask at all. The pipeline marks padding AND the dropped-DINO curriculum (dino_drop_prob=0.3, flow_heads.py:212-225) in a key mask; the model took a dense cond tensor and would have packed those rows as real tokens. Fixed the varlen way — masked tokens are simply NOT PACKED, which is also cheaper than masking. Verified: a sample whose DINO segment is dropped contributes 60 tokens where an unmasked one contributes 90.
• 🔴 concat_src coordinate mismatch passed silently. The concat is positional, so a mismatch would have trained texture against the wrong geometry with no error anywhere. Now asserted.
• 🟡 t_cond missed the std=0.02 timestep-embedder init (measured 0.044 — it kept the generic xavier). TRELLIS re-initialises every timestep embedder; ours now does too.
• 🟡 A fully-masked cond sample produced a zero-length segment, silently turning that sample unconditional. Now an explicit assert — unconditional is a state to ask for, not to arrive at.
• Process note: the first fix script aborted on an assert BEFORE writing, so two of the four fixes were never applied — caught only by re-running the verification (t_cond still read 0.044). A script that prints success is not evidence; the re-measurement is.

Verified by test, not by inspection

• Zero-init start: out layers and adaLN zeroed → the model outputs exactly 0 velocity at step 0 (TRELLIS’s own initialisation recipe).
• Varlen packing round-trips exactly with all three segments at different lengths AND different lengths per sample.
• Backward reaches 465/465 parameters; gradient checkpointing at block level (the block’s own use_checkpoint is dead here because we call its pieces rather than its forward).
• Every claim about a reference in this section was read from source: Modality Forcing at reference_repos/modality-forcing/flux_rgbd/, Hunyuan3D-2.1 at third_party_3d_gen/Hunyuan3D-2.1/. Three of my earlier statements about MF were wrong and are corrected above.

14.9 · Line-by-line audit (2026-08-15) — four parallel auditors + a full read

What the audit confirmed — measured, not asserted

• The block is TRELLIS.2's, op for op. _stream_pre+joint attention+_stream_post vs ModulatedSparseTransformerBlock._forward: max|diff| = 0.0 in float64. Modulation chunk order, which norm feeds the MLP, the rms-norm/rope placement inside QKV — all identical.
• Init matches structured_latent_flow.py's vanilla branch item for item; initial velocity is exactly 0.
• Packing round-trips bit-exact at three genuinely different per-sample lengths, with no cross-sample attention leak (verified row by row against cu_seqlens).
• tag_rows is an exact rotation group: four turns return to the identity bit-exactly; each turn is a swap and/or sign flip.
• Checkpointing is equivalent. Repeating the SAME configuration twice differs by 9.19e-3 relative; full-checkpoint vs no-checkpoint differs by 9.95e-3 — the same magnitude, so the gap is flash-attention's own atomicAdd backward nondeterminism, not the recompute. Forward is bit-identical.