Three Towers: folding sparse structure into one softmax
WilD3DGen v10 · weekly update · 2026-08-25
architecture3d-generationdiffusioneval
TL;DR
From-scratch failed for a mundane reason: v5–v8 were never warm-started. The config path was dead code.
v10 folds the sparse-structure tower (SS) into the same softmax as geometry and texture. No shared weights.
First eval where the model generates its own coordinates instead of being handed the ground truth.
1 · Why we changed course
We scaled capacity first. v7 0.8B → v8 3.19B. v8 passed v7's 10k-step best by step 4,330, then flattened in the same place.
Then we changed everything else — live conditioning, a 1.8M-asset pool at 36k, KD distillation at 50k, timestep schedule, adaptive clipping. Neither quality nor conditioning moved. Distillation pushed conditioning down.
Day six: the s3_*_t50b warm-start paths were never read. The plateau was undertraining, not a ceiling.
Quality and conditioning, same probe. Both sat still through the 1.8M pool and distillation — distillation pushed conditioning down. The warm start moved both in 2,000 steps.Capacity first, then data and distillation. Every attempt before day six was aimed at the wrong target.The warm start won in 1/25 the steps. That settled the recipe.
2 · What v10 is
Three towers, 3.9B parameters, all trainable, all warm-started from the s3 specialists.
Zero shared weights. They meet only in the attention.
SS is a third key segment, not a side branch and not a gated add.
3 · How SS is stitched in
Interleaved, not cascaded — the towers trade keys inside every one of the 30 blocks.
Masking is done by not concatenating. A zeroed key still takes attention mass, so the corner mask lives in the layout instead.
Stream identity rides on exact quarter turns; a borrowed key is re-roped into the other lattice.
Upstream hands off once, at the end. v10 meets at every block.The excluded segment is never gathered — not zeroed.3π/2 for SS, so both readers can separate every pair.Two ropes on one tensor. SS cell c sits between slat voxels 2c and 2c+1.
4 · Why the attention is dynamic
SS decides the token set the other two towers run on. Flip one SS cell and its block of slat tokens appears or vanishes.
SS is a constant 4096 tokens; the slat towers are sparse and asset-dependent. That asymmetry, not the parameter count, is what costs 3.4× the memory.
Flip a cell, and exactly its block moves. The attention row length moves with it.A different point set per asset. SS always gets the same full grid.One real batch: 4,496 to 16,324 keys. Nothing is padded.91% of assets have each slat tower smaller than SS.
5 · Results at checkpoint 1500
Every earlier number used ground-truth coordinates. This one does not. The ruler checks out: GT latent in → GT coordinates out, IoU 1.000 on 32/32.
Paired, n=32. SS did not move. Texture improved sharply — these 1500 steps repaired what the union did to the tex specialist.
14 up, 14 down, 4 unchanged. p = 0.75.Texture MSE fell 65%; std came back from 146% to 89%.What std 146% looks like. Same GT geometry in all three columns, so every difference is texture.The two cups look identical yet score 0.18 apart — occ_iou punishes sub-voxel alignment. The real failure is object identity: an armchair became a sofa.
6 · Open questions
No standalone-specialist control. So "is three towers worth it" has no evidence yet — only that it breaks nothing.
occ_iou needs a perceptual companion. The cup pair shows it cannot carry the headline.
Re-run this at 5k and 10k. If SS is still flat, the coupling needs rethinking.
