VLM-3D Stage-1 · ~27× speedup, 31.3% MFU (fla + sequence packing)

1 · What Stage-1 is

Autoregressive pretraining of Qwen3.5-2B (native hybrid VLM: full-attention + linear-attention gated-delta-net layers) to predict discrete 3D tokens — a representation-shaping pretrain (VP1). The AR head is dropped at inference; a flow model generates in Stage-2. Bidirectional multi-task: understanding (3D→text) + generation (image/text→3D), with multimodal replay data mixed in to preserve the base VLM (anti-forgetting).

• 3D tokenizer: 3 codebooks — SS (1728), shape (2048), tex (2048) ≈ 5824 codes/asset; vocab extended 248k→272k.
• Data: ~1.35M samples (gen_img 294k · gen_txt 293k · und 293k · replay_img 320k · replay_text 150k).
• Full-backbone training, DeepSpeed ZeRO-2, FlashAttention-3, 8× H200.

2 · Result — ~27× over naive, target MFU hit

The first two steps (FA3 + larger batch) are standard; the 3.6× jump from 8.6% → 31.3% MFU is the fla + packing work below.

3 · The winning recipe (8.6% → 31.3%)

1. transformers 5.9.0 — unlocks Qwen3.5 sequence packing (padding-free path requires ≥5.9).
2. Sequence packing (uniform 8192-token sequences) — kills the DDP straggler & padding waste from the huge length variance (467↔6882 tokens), and gives fla a finite set of shapes so its kernels compile once.
3. flash-linear-attention (fla) — fast-path kernels for the gated-delta-net layers; without it those layers fall back to a slow torch path (the real compute bottleneck).
4. Triton cache on node-local NVMe — fla's compile cache off Lustre; the default ~/.triton on Lustre causes 8-rank contention that stalls the compile (D-state hangs).
5. Clean GPU allocation — the node's keepalive holder yields only to a real Slurm job, not ssh/bash launches.

4 · The diagnostic journey (the lesson)

A clean per-step profile overturned the initial guess:

• The run was compute-bound, not data-bound. Tokenization is 9 ms; image preprocessing is hidden behind compute by prefetch (13× headroom). The data→NVMe migration treated the wrong problem.
• The compute was inflated by the missing fla fast-path (linear-attention torch fallback).
• fla alone (variable shapes) → recompile hell (15 s/it; Triton re-autotunes every shape). Packing → uniform shape → fla compiles a finite set once, then converges to 1.43 s/it.

5 · Takeaways

• Profile the actual training step (data vs compute) before optimizing — guessing sent the first effort at IO that was never the bottleneck.
• For hybrid linear-attention models, fla fast-path + uniform shapes (packing) are essential to reach high MFU.
• Keep compile/data caches on node-local NVMe; env-on-Lustre cold imports are a real startup tax and can D-state-hang.