| metric (8×H200, bs8) | baseline | after |
|---|---|---|
| true MFU (incl. attention) | ~10% | ~30% (SLAT) / ~31% (SS) |
| throughput | 1× | ~3× |
| per-GPU memory | — | ~105 GB / 141 (bs8) |
Four independent lossless levers, each verified: an fp32 segment-sum that removes a bf16 scatter-add that was 57% of the SLAT step; custom Triton fusion (LayerNorm+modulate, gate+residual) for the sparse flows; torch.compile for the dense SS flow; and FlashAttention-3 (Hopper-native) for attention.
| lever | where | gain |
|---|---|---|
| broadcast fix (fp32 segment-sum) | SLAT SparseTensor.__elemwise__ | 2.10× step |
| fused LayerNorm+modulate, gate+residual (Triton) | SLAT blocks | 1.15× step |
| torch.compile | SS (dense) blocks | 1.34× step |
| FlashAttention-3 (built from hopper/) | both (attention) | 1.10× step |
SLAT (sparse) can't torch.compile (dynamic shapes) → custom Triton; SS (dense) can't use the sparse kernels → torch.compile. FA3 fixes FlashAttention-2's ~35% Hopper utilization (no PyPI wheel → source build). All verified vs fp64 truth / loss A/B (Δ ≤ bf16 kernel variance).
| state | matmul-only | true MFU (with attention) |
|---|---|---|
| original (bf16 scatter-add) | 7.5% | ~10% |
| + broadcast fix (bs8) | 17.6% | 23.8% |
| + block fusion (bs8) | 19.8% | 26.8% |
| + FA3 — SLAT shape, bs8 | — | ~30% |
| + compile+FA3 — SS, bs2 | — | ~31% |
Precision = autocast bf16 (AMP), not full-bf16. GPU 100% busy; the residual wall is gemm (72%-efficient) + flash attention + cond/IO. FA3 underdelivers (1.10× not 1.5–2×) because few/short sequences starve occupancy — its full benefit needs bigger batch, which is memory-capped. ~30% ≈ the lossless ceiling here; past it needs fp8 (precision) or windowed attention / fewer blocks (model change).
| per-GPU memory (DeepSpeed ZeRO-1 + Adam + bf16, 8-GPU) | alloc |
|---|---|
| bs2 (old default) | ~38 GB |
| bs8 (new default) | ~105 GB (at elastic 0.75 target; 36 GB headroom) |
| bs12 | OOM (~147 GB w/o GC) — not worth it (heavier GC recompute) |
Wired into train_native_split.sh: PER_GPU_BS=8 (ga 4), ATTN_BACKEND=flash_attn_3, FUSED_MODULATE=1, --compile_ss_flow True (ss). Each behind a kill-switch / env override.