conditioning a frozen-geometry texture flow on a source-asset rail · 2026-07-30
editingTRELLISraildiagnosis
TL;DR
Instruction following: 0/6 → 6/6 rendered samples, 85.8% 10-way material top-1 (chance 10%). Two fixes did it: strip the instruction-blind DINO segment from the cond, and un-break the source rail.
The rail was silently disconnected for two full runs. Zero-init (the InstructPix2Pix recipe) left its columns at 2.4% of the pretrained scale after 30k steps — zero influence gives a near-zero gradient, a self-reinforcing dead zone that only escapes at 100k+ steps and large batch.
Still open: the model flattens spatial detail on every material (−0.152 variance ratio vs GT). Decals and surface patterns are not preserved.
1 · Architecture
Two channels, strictly separated. The VLM carries what to change; the source asset's continuous VAE latent carries what to keep.
Cond = instruction only. ~20 tokens, 100% instruction. This is the branch the base checkpoint already trains (its text-to-3D task), so the frozen connector stays in distribution.
Rail = channel concat. The tex flow already accepts a clean same-coords latent there (that is how the shape SLAT is fed), so widening one linear layer 64→97 is the only model change — loss, flow and sampler untouched.
Why channel concat is even legal: the data engine froze geometry, so source and target voxel coords match row-for-row (measured coord IoU 1.000, rail-flag hit rate 1.0).
Instruction and source travel on separate paths and meet only inside the tex flow — the rail never passes through the VLM, so it is free of quantisation loss.
2 · Channel vs token injection
Both are concatenation — the difference is the AXIS. Channel: rows stay, each row gets wider (64→97). Token: width stays, rows double (N→2N) and the output slices back to the first N.
Channel needs row-for-row coord alignment; frozen geometry gives us exactly that. It is unusable the moment the edit moves a part.
Token needs a stream code: sparse self-attention takes its RoPE phases from the voxel coords, so a source row and a target row at the same voxel are positionally identical and otherwise indistinguishable.
Measured cost is 1.29×, not 4× — attention is quadratic in rows but the linear layers dominate at our voxel counts (median ~1.7k).
Nobody has ablated the two for 3D object editing. Published work compares cross-attn vs token, or main-branch concat vs a side branch — never these two.
The same source latent, two axes. Channel concat is the cheap special case that only exists because the data engine froze the geometry.
3 · Failure 1 — the instruction was inaudible
v1 conditioned on [DINOv3(render) ; Qwen(render + instruction)], mirroring the base checkpoint's image path.
Token accounting: 2071 cond tokens, 15 of them the instruction (0.72%). The 1029 DINO tokens are frozen features that never saw the instruction.
Result: 0 of 6 samples followed the instruction. Consistent with an earlier tug-of-war probe on this stack — with DINO present, text leverage measured 0.0001; with DINO removed, 0.37.
Fix: drop DINO and the render entirely. Cond becomes the instruction alone.
The instruction was 0.72% of the conditioning signal. Removing the instruction-blind segments is what made the task learnable.
4 · Failure 2 — the rail was never connected
After the cond fix the model followed instructions but destroyed all source detail — and three inference-side remedies all failed.
The decisive check was not another sample: it was measuring the weights.
Rail columns sat at RMS 0.0022 against the pretrained shape columns' 0.0932. The model had never seen the source asset; it was re-painting textures from scratch.
This single fact explains every earlier observation, including why rail-dropout 0.3 vs 0.1 made no difference — dropping an unused input is a no-op.
Fix: initialise the new columns at the pretrained scale (std = 1/√97) and give only those columns a 10× learning rate via a gradient-slice hook. Pretrained columns verified bit-identical after widening (max|Δ| = 0).
Three inference knobs were tried and all were dead ends — the cause was upstream of all of them.Zero-init never escaped: 2.4% of the pretrained scale after 30k steps. Normal-init + 10× lr puts the rail at parity from step 1.
5 · Results
Instruction following is solved for this task: 6/6 rendered samples, 85.8% 10-way top-1.
Beating GT's own 74.7% separability is a warning, not a win — it means outputs are closer to the class prototype than real targets are, i.e. more stereotyped.
Spatial detail is short on all ten materials, worst on the materials whose GT keeps the most structure.
Likely cause: plain flow-matching MSE on a 32-d latent is dominated by getting the mean material right; pattern is a small residual with no gradient pressure behind it.
① Instruction landed. ② Every material under-preserves detail — a systematic bias, not per-asset noise.Final model. Colour and material land on all six; the van's decals (row 4) are still lost — the open problem.For contrast: the image-cond run, where the instruction was 0.72% of the cond. Nothing follows the instruction.
6 · The ground truth is not uniform
Measuring the data before blaming the model: GT's own spatial-variance ratio varies 2× across materials.
polished gold 0.61 — the engine genuinely flattens detail here, so a flat output is correct.
white marble 1.10 — the engine adds veining; flattening is wrong.
Consequence: 'does it preserve the pattern' has no single right answer — evaluation must be per-material against GT, which is how the chart above is built.
Two families inside one task: materials that flatten and materials that add texture. A single global preservation target would be wrong.
7 · Open questions
High-frequency detail. Add residual-magnitude loss weighting (w = 1 + α·|target − source| per voxel) so flat regions stop dominating the gradient — the analogue of the hard-example term used by published 3D editors.
This task is linguistically trivial. 10 distinct instructions, 1 template; the cond takes 10 values, so this is a 10-class conditional generator, not instruction following. Sibling tasks in the same dataset carry 2,840 unique instructions.
Channel vs token injection is still unmeasured. No published 3D-object-editing work ablates the two; the token arm here needs the dual-timestep modulation the reference implementation uses (clean source tokens must be modulated at t=0) before its number counts.
Structural editing needs part grounding, which a text-only cond cannot provide — that is where feeding the asset to the VLM as discrete 3D tokens becomes necessary.
single seed · 21,859 forward pairs · frozen geometry · warm-started from a 40k-step generation checkpoint · all numbers measured, none estimated
