Why do our generations look oily? A four-layer material audit

0 · TL;DR

So this is not a code bug to patch; it is an out-of-distribution material-decomposition failure. The lead explanation is a domain gap we measured a month ago and never closed: every conditioning render in training is flat-lit (no specular, no environment reflection), while every real photograph is not.

1 · Symptom

The deployed demo (image → SS 64³ → shape SLAT-512 → tex SLAT-512 → GLB) produced results that read as wet plastic / polished chrome. Reading the material of one generated GLB:

metallic ≈ 1 with roughness ≈ 0.4 under an environment map is exactly the "oily film" look: every surface mirrors the environment.

2 · Four-layer audit — every layer is clean

Rather than guess, each stage was measured against ground truth on real assets.

3 · In-distribution fidelity — the model is not globally biased

Eight assets, teacher-forced (GT coords + GT shape so the geometry stages cannot be blamed), predicted material vs each asset's own GT:

Seven of eight assets are faithful to within 0.03. One asset (38a5b1fb) flips to 0.437 — so the failure mode exists in-distribution but is rare. Note the roughness column is noisier than metallic (e.g. 0.993 → 0.668), i.e. the model is less certain about gloss than about metalness.

4 · Refuted: it is NOT exposure or specular highlights

The intuitive explanation — real photos carry baked-in lighting and specular highlights, which get misread as metal — is exactly the failure Seed3D 2.0 names ("specular highlights on non-metal surfaces misclassified as metallic responses"). We tested it directly: one training render, one property perturbed at a time, same seed, teacher-forced.

5 · What actually separates the two regimes

Grouping every GLB the demo has produced by input type:

The split is clean and it is not a code-version effect: one of the high-metallic runs predates the most recent demo rebuild, i.e. the same binary produces ≈0.00 on renders and ≈0.5–1.0 on photos.

6 · Where the field is, and what we propose

Reading the three current systems, material realism is a main axis of competition, and one of them states our exact failure mode and its fix.

The strongest remaining hypothesis: our conditioning renders are flat-lit

Mining the project history surfaced a measurement from 2026-06-25 that reframes this whole audit. Checking renders_cond against a known-metal asset (an ABO step ladder) showed the conditioning renders carry only soft grey gradients — no sharp specular, no environment reflection. Metal is rendered as matte grey; a captioner calls it "metal" from shape and commonsense, not from material cues. The conclusion recorded at the time was that these renders are unsuitable for material work — and the rig was never changed.

This also explains the asymmetry in Seed3D 2.0's setup: they render training views under random HDRI so highlights and reflections are visible, and then add VLM material tokens to disambiguate what those highlights mean. We have neither half. Two concrete levers follow, in dependency order:

1. HDRI-lit conditioning renders (or photo-domain augmentation) so specular and reflection appear in training at all — without this the model has no basis for separating "shiny dielectric" from "metal".
2. Material-semantic conditioning from the VLM, to resolve the residual ambiguity that lighting alone cannot.

Proposed direction: VLM-native material semantics

Material decomposition from a single photo is ill-posed — a white highlight can be metal or a lit dielectric, and pixels alone cannot separate them. The disambiguating signal is semantic: knowing the object is a wooden chair fixes metallic ≈ 0. Seed3D 2.0 validates this route with a light touch (a few VLM description tokens). Our architecture conditions on a VLM throughout, so the same idea can go further:

• Verified today: our 2B VLM already describes material and structure correctly from a framed render — asked about an unlabelled asset it returned "a low-poly architectural model … white exterior walls … blue patterned facade … open top revealing the floor plan … matte finish rather than glossy", every clause checkable against the image. What it does not do is map "matte" to PBR values or emit a stable schema.
• Data already exists: a 1.10M-caption texture pass already exists with a structured schema (global_material{metalness, reflectivity, transparency}, per-part {material, metalness, roughness, finish}). Correction to an earlier draft of this report: those captions are qualitative and produced from the same flat renders — the GT-grounded variant (decode pbr_latent → k-means material regions → feed measured metallic/roughness to the captioner) was prototyped on 9 assets and never run at scale. So the (material words ↔ real PBR values) supervision does not exist yet; it is a ~2-day 8-GPU pass on top of an already-validated prototype.
• Inference cost is negligible: ~60 generated tokens on a 2B model ≈ 1–2 s against a 30–60 s cascade, and the emitted description is human-readable, so it can be shown and edited in the UI — a control surface none of the three systems expose.