This report consolidates the entire arc — behavioral nulls, the mechanism probe, the full VGGT-REPA sub-investigation (3 more nulls), and the sincos ablation that finally moved the mechanism — with every visualization produced along the way. Supersedes the earlier multi-image-ablations and SS-REPA v1 reports.
Deployed stack: a 3-stage rectified-flow cascade — SS (16³ occupancy) → shape SLAT-512 → tex SLAT-512 — conditioned by cross-attention on [DINOv3 spatial tokens ; connector(VLM hidden)]. The multi-image (IM) path packs 4 views into one cond: a joint-4-view qwen segment + 4 per-view DINO blocks carrying per-token view_ids. Experiments target the SS stage (coarsest, cheapest to iterate) on 14 held-out assets (0-leak verified), IoU@64, official sampler; node-local MDS + compile-off fixed a Lustre data-starvation straggler (8-12 → 2.5 s/it) so training was viable at all.
Question: does 4-view conditioning give better 3D than 1 view? Acceptance: (a) 4-view IoU > 1-view IoU; (b) the model uses distinct view content, not just "4 image slots".
Visibility analysis (z-buffer splat on GT surface voxels, silhouette-validated vs the actual renders): best single view sees 49% of observable surface; the 4-view union sees 77% — views 2-4 add +28pts of genuinely new surface, positive on all 14 assets. Redundancy hypothesis falsified: the data carries it.
But the model discards it. The sharp diagnostic — 4 distinct views vs the same view copied 4× — is a dead heat: 4-distinct 0.383 ≈ 4-copy 0.385 ≈ 1-view 0.384. The model conditions on "4 image slots", not on their content.
An attention probe (explicit fp32 softmax, per-voxel attention over the 4 view blocks, validated row-sums) contrasts the SS flow with the shape flow. SS is fully content-blind: per-voxel argmax-view is one fixed slot everywhere, bit-identical between 4-distinct and 4-copy. The shape flow shows genuine, if weak, visibility-aligned routing in mid/deep blocks — different sides of the object attend the views that see them (teacher-forced 4d<4c on 14/14, though the effect is ~0.5% of loss). This localizes any real multi-view fusion to the FINE stages, not coarse SS.
| # | intervention | result |
|---|---|---|
| 1 | Train fusion on 4-view cond (IM probe) | ❌ ignores content (4d≈4c) |
| 2 | View-count randomization (1–4/sample, TRELLIS recipe) | ❌ good-view band → 1 view still suffices → no pressure |
| 3 | TRELLIS-1 inference aggregation (multidiffusion / stochastic) | ❌ 0.398 / 0.390 < single 0.404; averaging conflicting views crashes view-inconsistent assets |
| 4 | VGGT-REPA v1 (32-d target, random projection) | ❌ cos plateau 0.12 (target gutted) |
| 5 | VGGT-REPA v2 (raw 2048-d target) | ❌ cos plateau 0.057; downstream null on all 3 metrics vs matched control |
| 6 | VGGT-REPA v3 (λ=10 warmup, early ckpt, tap block 5) | ❌ same cos ~0.05 wall — λ×5 doesn't break it |
The VGGT target itself was a rabbit hole worth noting: VGGT has no fused-3D latent — it emits per-frame 2D patch tokens and reprojects 3D per-view. Its self-estimated cameras could not register our object-only renders under any sim(3) (residual ~0.9 of extent), so the target had to be built via a "gtcam" lift (per-view pointmap placed with the GT camera). That gate passed at 16³ (coverage 0.91, fusion signal +0.26 in the target) but the voxel↔patch correspondence still carried reprojection noise — a second reason the alignment couldn't exceed the mean.
The turning point was questioning the view-identity signal itself. Our per-view embedding (dino_view_embed) was zero-init learned — it trains to only L2≈2, a weak slot tag. Hunyuan3D-2-mv (and our own earlier dpos position-stamp fix) instead use a fixed full-strength sincos encoding. Hypothesis: routing never formed because the "which view" signal was too weak from step 0. Three matched-step runs, only the view-embed differs:
| view-embed | attn top1 (routing) | DINO attn share | 4d−4c gap | abs IoU |
|---|---|---|---|---|
| zero-init (L2≈2) | 0.53 (= random) | 0.72 | +0.001 | 0.383 |
| full sincos (L2 22.6 ≈ 0.7× DINO) | 0.62 (routing ON) | 0.22 COLLAPSED | −0.025 (worse) | 0.293 (crashed) |
| scaled sincos (L2 4.5 ≈ 15% DINO) | 0.65 | 0.64 (preserved) | +0.022 | 0.389 |
Zero-init too weak → routing = random. Full-strength sincos creates routing but drowns content — the flow flees 78% of attention to the qwen segment (DINO share 0.72→0.22) and geometry crashes. Scaling to ~15% of the DINO norm gives a clear view signal without out-weighing content: routing forms AND the DINO share survives, and 4d−4c finally turns positive.
Scaled-sincos trained to 10k. The routing win is real and stable but saturated, and never converts to occupancy quality:
| metric | 2k | 4k | 10k | zero-init |
|---|---|---|---|---|
| 4d−4c gap | +0.0245 | +0.0232 | +0.0224 | +0.001 |
| attn top1 | 0.631 | 0.632 | 0.633 | 0.527 |
| abs 4-view IoU@64 | — | — | 0.389 | 0.383 (I1=0.404) |
The full-cascade render eval (SS→shape→tex→GLB, 4-view vs 1-view) tells the same story visually: on the assets that reconstruct, the textured 3D is essentially identical between 4-view and 1-view.
Verdict — multi-view at the SS level is a dead end, and we know exactly why. Six interventions null; the seventh (scaled sincos) is a genuine mechanistic win (content-blind → visibility-aligned routing) that saturates at a +0.02 contrast with no absolute gain. The bottleneck is not view-signal strength, architecture capacity, or teacher quality — it is the absence of fusion pressure in a coarse-occupancy objective.
What would actually help (not pursued — low ROI vs the validated single-image 0.40 and text 96%-of-GT lines): (a) force fusion where single views are physically insufficient — full-sphere / occluded / bad views in training, judged at the shape/tex level where multi-view surface detail is representable (shape already shows weak real routing, §3); (b) per-token view signal on the qwen segment too (needs a cache rebuild). Both change the incentive/representation, not just the signal.
Deliverable value: a clean ablation (zero-init too weak < full-sincos collapses share < scaled-sincos balanced), a mechanistic interpretability result (attention routing responds to view-embed magnitude), a decisive negative (routing does not imply geometry under a fusion-free objective), and validated infra (node-local MDS 2.5 s/it, sincos/dpos-style view encoding).