ready_v4_vlm_filtered set, warm-started from the single-image production cascade. Three stages (SS / shape-SLAT / tex-SLAT) train in parallel, one per GPU node.This report is the config of record for that run: what the data is, what the model is, and — importantly — the four setup bugs that were found and fixed before the numbers can mean anything. Results are reported separately.
| task | weight | manifest | cached cond root | cond shape |
|---|---|---|---|---|
image_to_3d | 0.5 | vlm_filtered_all.jsonl 419,999 → 404,412 after aesthetic≥4.5 | v22_3dvlm_tok1024_mv1 (v000) | qwen (1054, 2048) + DINO (1029, 1024) |
multi_image_to_3d | 0.3 | same | v22_im4r (m00) — rebuilt | qwen (292, 2048) + DINO (1620, 1024) + view ids |
text_to_3d_weighted | 0.2 | vlm_filtered_capT.jsonl 422,039 → 406,427 (caption required) | v22_3dvlm_tok1024_mv1 (t000..t003) | qwen only (~56-93, 2048), no DINO |
Sampling is batch-granular: one task per batch, so padding is to the batch's own longest sequence — a text batch never pays for the multi-view budget. Measured task mix over 40 batches: 0.500 / 0.300 / 0.200, exactly the weights.
The IM cache was rebuilt (1.90 TB, 62 min on 24×H200). The previous cache pinned each asset to 4 views drawn from the "good band" only (elevation 005-011). Real user inputs are not from a good band, and — more importantly — four similar views give the model no reason to fuse. The rebuild samples 4 distinct views out of all 16 with weights biased toward usable angles:
| elevation band | weight | share of sampled views (8k-asset audit) |
|---|---|---|
| 000-002 below-ground | 0.15 | 4.65% |
| 003-004 low | 0.6 | 11.96% |
| 005-011 eye-level..3/4 | 1.0 | 63.39% |
| 012-015 high / top-down | 0.5 | 20.01% |
Deterministic per-sha (seeded by the sha), so the build is reproducible and resumable; the chosen indices are stored in each entry for audit. 419,996 / 419,999 assets built (3 have broken source renders).
The text cache was built the same session: 1,688,136 caption entries (422k assets × 4 captions), 352 GB, ~10 min on 24 GPUs, written beside the image entries in the shared root.
On the previous cluster the per-asset npz read pattern starved the GPUs (8-12 s/it stragglers), and node-local MDS shards were the fix. That conclusion is cluster-specific and does not transfer. Measured on the new FSx Lustre, with the real training dataset and DataLoader (npz decode + normalize + collate), cold reads over disjoint file sets:
| dataloader workers | throughput |
|---|---|
| 0 | 31.0 samples/s |
| 4 | 134.9 samples/s |
| 8 | 290.2 samples/s |
| 16 | 731.1 samples/s |
| knob | value | why |
|---|---|---|
connector (cond_adapter) | xf2 — i1-style 2-block Transformer, 27.3M | replaces the 3M MLP. Measured negative on qwen-only SS IoU earlier; kept here on the bet that the extra capacity pays off for text-to-3D, which this run can finally test |
| view embedding | sincos, scale 0.2 (fixed buffer, DINO segment) | zero-init learned embeddings stay a weak slot tag and never enable routing; full-strength sincos (L2 22.6 ≈ 0.7×DINO) drowns the content. 0.2 → L2 4.525 ≈ 14% of the DINO token norm (32.0), the ratio that worked |
pos_stamp (dpos) | off | it is a no-op for IM and T anyway (guard fires below 1024 qwen tokens; IM has 292, T ~93), and on I1 DINO already carries position. Only cost is the DINO-dropped inference path |
| view-count randomization | on — 2..4 view blocks per IM sample | see bug 1 below |
| DINO dropout | 0.3 | per-sample, whole segment masked off |
| qwen dropout | new, mutually exclusive with the DINO drop | without it the flow can escape to the qwen segment whenever DINO gets harder to read — the exact failure seen when a strong view embed collapsed DINO's attention share 0.72 → 0.22. Text batches are untouched (qwen is their only cond) |
| batch | eff 256 = bs8 × 8 GPU × ga4, per stage | matches the previous cluster's setting, so numbers stay comparable |
| warm start | fusion_ss_dpos_2n@22000 · fusion_shape_v22gv@16000 · fusion_tex_v22gv@16000 | the single-image production cascade |
| schedule | 3000 steps, 3 nodes in parallel (one stage each) | SS 3.82 s/it, shape/tex 3.39 s/it — ~3 h |
Measured token norms that set the embedding scale: qwen hidden L2 ≈ 165-169 (2048-d), DINO L2 = 32.00 (1024-d). The connector ends in a LayerNorm, so its 1024-d output lands at the same magnitude as DINO — one scale knob serves both segments.
n_views ∈ [2,4] per batch, but then clamps it to a cached combo size — and the cache ships only im_combo_sizes=[4]. The clamp made the draw a no-op: every IM sample saw all four views, which is precisely the measured root cause of zero fusion pressure (4-distinct ≈ 4-copy ≈ 1-view). Fixed by subsetting the loaded combo down to the drawn count, dropping whole view blocks via dino_view_ids. Verified live: view counts now come out {2: 32, 3: 16} instead of all-4.--cond_adapter from the command let the default (xf2) build while the checkpoint held an MLP connector: missing=35, unexpected=4. The load prints those counts — worth reading every time, since the run otherwise looks perfectly healthy.num_workers > 1 may interleave tasks and break batch homogeneity. Measured at 1 and 12 workers: 0/25 mixed batches either way — PyTorch assembles each batch inside a single worker for iterable datasets. 12 workers kept.Also fixed: the eval loader hardcoded the MLP connector and would have mis-loaded every xf2 checkpoint; it now picks the architecture from the checkpoint's own keys.
The view embedding is applied to the DINO segment only. In the multi-view case the qwen segment is a joint encoding of all four images (292 tokens, 64 per view after a 7-token prefix, verified constant across the whole cache), and nothing tells the model which token belongs to which image, or where inside that image it sits.
Designed but not yet implemented: a factorized code on the qwen image tokens, in disjoint dimension bands so the axes stay independently decodable — view ordinal in one band (shared by all 64 tokens of an image), then row and column of the 8×8 grid in the other two. Summing full-width codes instead would superimpose them and force the model to disentangle; disjoint bands cost nothing and avoid the question. Text tokens get no code.