WilD3DGen · Weekly update, Sep 1 – Sep 8

TL;DR

1 · Text-to-3D benchmark closed out (Sep 1–2)

• runs/s3_ss_roadv1 (ROAD-REPA SS, 60k × 256) finished; EMA became the standard eval weight after raw-weight snapshots were shown to oscillate (28k "clearly better" vs 33k "mediocre" tracked the same seed-0 samples).
• Text track (eval_suite/text/, 47 SWH prompts, 0/47 in TRELLIS-500K) scored with certified scorers (Uni3D-g 87.3 / ULIP-2 73.2 top-1 on ModelNet40).
• UV bug: trimesh flips glTF V to a lower-left origin, nvdiffrast expects upper-left — every atlas rendered as scrambled patches. All render-based metrics before Sep 2 are void; new GLB sources must pass uv_flip_probe.py.
• After the fix the earlier "roadv1 improves appearance (KD 48→43)" claim reversed; the 2.2× appearance gap to the baselines sits in the tex tower, not in SS.

• Paired tests: SS-swap (v10 SS → roadv1 SS, same towers) CLIP-T +0.014 (t=3.1), ULIP-T +0.019 (t=3.0), Uni3D-T +0.031 (t=4.1); tower-swap with fixed SS |t| ≤ 1. 60k vs 46k: no significant change.

2 · Frozen-DiT ablation and the v12 recipe (Sep 2–4)

• runs/s3_ss_roadfrz: roadv1 recipe with the DiT frozen (--flow_tune none, 36.7M trainable), 60k schedule, stopped at 35k. Connector-only reaches the same occupancy IoU as the fully fine-tuned roadv1 at 46k (text 0.110 vs 0.109) and higher ULIP alignment at 20k (0.995 vs 0.931) — the connector, not the DiT, carries most of the early gain.
• Literature/config audit (TRELLIS, TRELLIS.2, TIGON, Pixal3D): all use constant LR and EMA 0.9999 → new runs use EMA 0.9999 with warm-up; WSD schedule with forked 6k decays.
• v12 plan locked: S1 connector-only 20k+6k @1e-4 → S2 full unfreeze 100k+6k @2e-5 (connector and DiT same LR) → S3 unify 30–40k @1e-5. Mixture per tower: 60% real text / 10% synthetic text / 15% single-image / 15% multi-image (2–4 views, DINO 512).
• Conditioning (CondStamp): hashed near-orthogonal view codes (unlimited views), fixed segment codes for DINO/Qwen streams, fixed 2D sin-cos patch positions for Qwen image tokens. Geometry-tower ROAD implemented (Uni3D teacher, tap at block 9).
• Eval hygiene: 23,456 eval/held-out assets removed from every training pool (*_train_evalclean.jsonl); roadv1/roadfrz/v10 remain contaminated on the SWH track.

3 · Texture-swap bug in renders_cond: audit, re-render, re-caption (Sep 4–6)

• Found from a tex-tower probe: the decoded GT texture did not match the condition render for two TexVerse assets. A 100-asset human audit found 15–20% of TexVerse wrong, all with a texture belonging to a different asset.
• Root cause: blender_batch_cond.py renders 25 assets per Blender 3.0.1 process with use_persistent_data=True; Cycles caches images by data-block pointer, so after init_scene() frees them, a later asset reuses an earlier asset's pixels (Blender T92980). Reproduced: persistent ON 25/43 wrong, OFF 0/43. The first asset of every chunk is never affected (0% at position 0, 41% at position 5+). pbr latents are correct; captions, VLM scores and aesthetic scores are contaminated.
• Full-pool scan (96 GPU workers, ~2 h): chunk order rebuilt for 862k assets; each render compared to its own decoded GT and to the GT of same-chunk predecessors with a lighting-robust 32×32 chromaticity histogram. SWAP if a predecessor's GT matches better; MISMATCH if far from own GT; plus a strict p85 rule (user's choice: "rather re-render good assets than let a bad one through") and a pixel-level rule for assets without chunk order.
• Tally: 281,900 confirmed bad → 461,094 flagged with strict rules; 286k re-renderable (source GLBs on hand), ~153k excluded only (SWH / ObjaverseXL sources unavailable). Eval sets were contaminated too (val200 36, test200 29, sketchfab_val200 45, sketchfab_test200 51) and were re-rendered first.
• Re-render: one Blender process per asset, persistent ON (verified pixel-identical to OFF, 1.7× faster); 6 nodes × 16 workers, 11–13k assets/h, CPU-bound (Blender's GPU utilisation 5–45%). 313,194 assets re-rendered into renders_cond_refix in ~26 h.
• Acceptance: new render vs GT still mismatched for TexVerse 9.4% and SketchfabV1 24.3% — traced to the colour-space convention below, not to rendering.
• Re-caption: Qwen3.6-27B holistic + texture captions for 309,337 re-rendered assets (48 vLLM shards, ~1.5 h per pass, parse failures 16 / 618k).

4 · Second finding: flat-colour materials stored in linear light (Sep 6)

• Acceptance leftovers were all textureless materials. o-voxel/…/volumetic_attr.py:458 writes textured colours as sRGB bytes but writes baseColorFactor raw (glTF spec: factor is linear, texture is sRGB) — so flat-coloured parts decode darker and more saturated than any renderer shows them (orange input → deep-red GT).
• Scale (source glTF header scan): SketchfabV1 16.5% all-flat + 7.1% mixed; TexVerse 0.9% + 19.9% mixed. Textured materials are unaffected.
• Official TRELLIS.2-4B reproduces the bias (4/5 coloured all-flat assets generate too dark/saturated); no GitHub issue exists — it is a toolkit convention, not our bug.
• Fix (user's tier A): decode → linear→sRGB on base colour → re-encode with the frozen tex encoder (coords identical, 8/8 pilot exact). All-flat assets fixed in full (68k); mixed assets fixed per material cluster with a render judge (63k; validation recall 77%, 13% partial false edits on textured assets). SWH/ObjaverseXL (no sources) left as-is. 97,192 repaired latents in trellis2/_flatfix/.
• Revertible by construction: no file moved. *_train_texfixed.jsonl points renders/captions at the re-rendered set and drops non-re-renderable bad assets; *_train_texfixed_ff.jsonl additionally points pbr latents at the repaired ones. Revert = switch manifest.

5 · v12 Stage 1 on fixed data (Sep 6–7)

• v12s1b: three towers from scratch (init none) on the _texfixed_ff mixes; connector-only, LR 1e-4, 20k stable + 6k cosine, eff. batch 256, 2 nodes per tower. SS 2.2–2.6 s/it (ROAD), shape/tex 1.7–1.9 s/it. All three reached 26k by Sep 7 midday.
• Shape image path improved monotonically through S1 (surf_dev 0.057 → 0.028, already below the t50b reference at 0.050). Tex image path plateaued at ~0.23 vs t50b 0.15 — a frozen DiT cannot fix appearance. Text paths of both towers stayed 2–3× worse than image paths.
• SS at 5k on fixed data vs the same recipe on old data: ULIP text 0.860 vs 0.804, image 0.997 vs 0.967 — the only same-step comparison we have of clean vs dirty data.
• Held-out n=16/24 probes swing ±0.02 between checkpoints; trends are read from ≥3 points, single checkpoints are not conclusions.

6 · v12 Stage 2 full fine-tune (Sep 7–8, running)

• v12s2: init from S1 checkpoint-20000_ema (EMA overlaid onto model.safetensors, since init only reads that file), --flow_tune full, LR 2e-5, 100k stable + 6k decay, save every 5k (limit 6), EMA 0.9999. Shape/tex bs8 × GA2 with elastic GC 0.6 (0.75 OOMed), 2.4–2.6 s/it; SS bs4 × GA4 without GC (ROAD hook incompatible with GC; tap-safe GC fits but is 4–5× slower), 3.05–3.5 s/it.
• Status Sep 8 23:00: shape 45k, tex 45k (resumed from 40k after the quota failure), SS 29k. ETA to 106k: shape/tex ~Sep 10, SS ~Sep 11; the 60k fork-decay checkpoint is due in ~12 h.
• Readings so far (EMA): tex image reaches t50b parity by S2 10k and holds (0.135–0.154 vs 0.150); shape image 0.037–0.063 around the 0.050 reference, but val32 shows 0.083 vs 0.063 at 40k (single point); SS ULIP text 0.896 → 0.975 with retrieval top-1 0.333; occupancy IoU flat (text ~0.10, image ~0.22). Text paths of shape/tex are not yet better than the S1 end.

7 · Incidents and lessons

• Disk: /fsx/home hit 100% (quota 39.65/40 TB) during the 15k saves — 2.2 TB of TexVerse re-download copies deleted, save_total_limit 24 → 6, half-written checkpoints quarantined. Then the user quota was cut to 5 TB (37.8 TB used): tex lost its 45k save; SS/shape will fail at 30k/50k unless resolved.
• OOM: shape at 1610 (GC 0.75 → 0.6), tex at 11.5k (→ 0.6), SS at start (bs8 → bs4×GA4). GC experiment on SS cost 2.5 h.
• Shared tmux server on the agent node restarted and silently killed the reservation heartbeat and two towers' srun clients; the benchmark service then took the SS GPUs. Launchers moved to node-local tmux per rank; heartbeat now self-restarts from the monitor.
• Hold rotation: SS died on a 7-day hold timeout and waited 3 h for permission to restart; rule now: re-point to the new hold and restart without asking. Likewise S2 waited 4 h for a go-ahead that the plan already gave.
• pkill -f inside srun bash -c kills its own step (three times this week) — kills now go through script files.
• Feeder crash on an unknown subset idled 96 render workers for 12 h overnight; fixed with subset skip, HF timeouts and orphan reclaim.
• Agent shells (claude/codex) moved from the login node to compute nodes; cluster skill updated (§0.1.2).

Open questions

• Quota: restore 40 TB or shrink to 5 TB (would need deleting ~33 TB). Candidates for cleanup: runs/s3_*_v12s1 (150 GB), S1b intermediate checkpoints (~400 GB), s3_ss_roadfrz (45 GB), runs/_broken.
• Is the shape image-path gap on val32 (0.083 vs 0.063) a trend? Needs the 50k point; formal val200 evaluation at the 60k fork.
• Will the text paths of shape/tex recover during S2, or does text need its own fix (caption weighting, more epochs)?
• Flat-colour convention on SWH/ObjaverseXL (no sources) — leave, or re-voxelise from decoded slats? File a TRELLIS.2 issue?
• ~153k non-re-renderable swapped assets remain excluded; ~307k assets without GT were never judged.
• No ablation yet of fixed vs unfixed data at equal steps beyond the SS 5k point.