← Back to project

Molmo2 Codec V6 — Stage-1 & Stage-2 Training Insights

What was built, what failed, and what the 8K counterfactual pilot actually proves · 2026-07-31
V6AdaCodecStage 1Stage 28K budgetcounterfactualMLVU

1 · Executive readout

Stage 1
P content is real
At n=16, matched P lowers held-out reconstruction loss by 24.2% vs zero P and 25.3% vs foreign P.
Ordinary Stage 2
CE can ignore P
Dense 8K retention stays flat while codec accuracy drops 4.0 points by step 800.
Repair
+7 points in 20 steps
Short real-P accuracy rises 58.0→65.0%; strict P-content attribution is still pending.
Current interpretation: V6 Stage 1 encodes useful video-specific information, but standard QA supervision does not force the LLM to consume it. A P-dependence objective is the promising direction; the donor corruption still needs a strict content gate.

2 · End-to-end V6 design

Stage 1 · learn a compact predictive visual code full 2-fps timelineadaptive GOPI + 1…16 P V6 P-tokenizeru5 → copied P-ViT144 latents → connector16 tokens / P-frame Hφ auxiliary predictorreads zI + zP1…zPnpredicts normalized deltadiscarded after Stage 1 frozen native teacherzT = RGB target featureL1 + cosineI path remains unchanged train visual Pfreeze I + LLMGBS 128 Stage 2 · teach the frozen codec interface to the LLM 50% native videoup to 101 RGB frames≤8,181 visual tokens 50% codec videocomplete GOPs under 8,19216 P / 81 I tokens dynamic packing16,384 total contextsample-isolated masksformal GBS 128 Molmo2-4B LLMvisual side frozenordinary answer CE+ optional P-dependence loss paired gatesnative retention · dense vs codecreal vs zero/foreign/shuffle Pcorrect-answer log probability

3 · Stage-1 objective

V6 predicts the target-frame change, then reconstructs the native feature reference zIfrozen Molmo feature81 native tokens P sequence zPn × 16 compact tokensn = 1…16 Hφ(zI, zP)predict d-hatd = (zT − zI) / sposspos learned per position reconstruct targetzT-hat = zI + spos · d-hatteacher zT comes from RGBsame frozen I visual path Stage-1 lossmean |d-hat − d|+ 1 − cosine(d-hat, d)Hφ is discarded after training Important: direct cosine(P token, RGB token) is not the reconstruction metric; the token sets have different semantics and widths.
ComponentV6 implementationTraining state
I-frame encoder + connectorOriginal Molmo2 vision path · 81 tokens/framefrozen
P-frame inputRGB residual + 2-D motion · signed 5-channel u5fixed codec construction
P visual encoderCopied Molmo ViT · 144 learned latent queries · split block masktrain
P connectorIndependent copied native 3×3 attention pooling + MLP · 12×12→4×4train
Temporal codesinusoid(t) × learned scale + learned residual · t=1…16train
HφPredicts position-normalized target delta from zI + all preceding zPtrain, then discard
Formal runrandom n=1…16 · frame budget 128 · GBS 128 · BF16 · 8 H200step 17K checkpoint evaluated

4 · Stage-1 reconstruction evidence

Held-out controls: the same Hφ sees matched P, all-zero P, or P from a different held-out example. Lower loss for matched P is the direct test that the compact tokens carry sample-specific predictive information.
The matched-P advantage is nearly absent at step 1K and becomes large by 5K–17K; Stage 1 learns content rather than only a reference-frame prior.
At step 17K, matched P stays near 1.17 loss while zero/foreign controls worsen with longer GOPs; the 16-P regime is not the weak point.

5 · Stage-1 metrics and representation geometry

MetricMeasured valueMeaning
n=16 matched-P held-out loss1.1693delta-position L1 + cosine distance
n=16 zero-P loss1.5429matched P reduces loss by 0.3736 / 24.2%
n=16 foreign-P loss1.5661matched P reduces loss by 0.3968 / 25.3%
n=16 reconstructed raw-feature L12.7047native feature units; not a normalized percentage
late minibatch cosine similarity≈0.52–0.58logged cosine term is 1−similarity; reconstruction is useful, not near-lossless
direct aligned P/native token cosine−0.0014raw P tokens are a code, not native RGB tokens; this is not a reconstruction score
cos(full P, residual-only)0.938residual is the dominant content carrier
cos(full P, motion-only)0.701motion is non-trivial but not sufficient alone
cos(full P, no-time)0.989temporal embedding perturbs less than visual content
On the exact MLVU n=200 zero-shot boundary probe, Hφ matches native RGB accuracy while deployable P16 trails by 3 points at 26.4% of the visual-token count.

6 · 8K mixed Stage-2 contract

Full timeline first; token budget second decode at 2 fpsup to 2,048 frames≤1,024 secondsbefore sparsification adaptive GOPI/P roles from predictive costmotion ±48 · 14×14 blockscomplete timeline contract select complete GOPsuniform temporal coverage≤8,192 visual tokensno partial-GOP truncation representation-matched twinscodec: 81/I + 16/Pnative: 81/frame, max 101same source mixture · 50/50 Training poolstratified 80K-video pilot manifest · long-codec eligibility: ≥4K selected tokens and ≥128 P frames · 10,979 eligible examplesRuntime16,384 context · dynamic packing · GBS 128 · frozen visual side · cache-only GOP artifacts
ContractValueWhy it matters
Visual budget8,192 tokensreplaces the underpowered 1,377-token pilot
Total context16,384 tokensleaves space for frame markers, prompt, and answer
Representation mix50% codec video / 50% native videoisolates codec adaptation and native retention; no image branch in this pilot
Timeline2 fps · max 2,048 framesup to 17.1 minutes before GOP selection
Codec long-video floor≥4,096 selected tokens · ≥128 P framesprevents nominal “8K” samples from collapsing to short clips
Pilot data80K stratified video manifestdiagnostic subset, not the final 887K-example corpus
OptimizationLLM-only · GBS 128 · 100-step pilot warmupkeeps the visual code fixed while testing learnability

7 · Stage-2 result curve

Standard CE can preserve dense behavior while degrading codec behavior. Counterfactual repair quickly reverses the trend, but the long codec score only returns to its zero-shot baseline.

8 · Exact Stage-2 evaluation log

CheckpointShort real P16Short Hφ818K dense8K codec
untuned LLM / step 058.5%61.5%77.0%72.5%
ordinary QA step 40054.5%56.5%——
ordinary QA step 60058.5%57.5%——
ordinary QA step 80058.0%57.0%77.0%68.5%
counterfactual repair step 82065.0%62.0%80.5%72.5%
Paired comparisonNet changeExact evidence
step 820 real P vs step 0+13 / 200 · +6.5 points28 wrong→right · 15 right→wrong · McNemar p=0.066
step 820 real P vs Hφ+6 / 200 · +3.0 points16 P-only correct · 10 Hφ-only correct · p=0.327
step 820 8K codec vs step 00 / 200 · 0.0 points15 wrong→right · 15 right→wrong
step 820 8K dense vs step 0+7 / 200 · +3.5 points19 wrong→right · 12 right→wrong
Evidence boundary: the accuracy recovery is not P-specific. The strict content gate below shows that real P does not beat foreign or shuffled P with the weak within-pack corruption recipe.

9 · Counterfactual repair and current caveat

Make correct answers depend on matched P content real branchquestion + I + matched Plog p(correct | real) corrupt branchsame question + I + wrong Plog p(correct | corrupt) answer-token gapg = mean[log p(real)− log p(corrupt)]eligible temporal codec QA only repair objectiveL = CE + ReLU(0.1 − g)weight = 1.0visual modules stay frozen strict evaluationreal vs zero Preal vs foreign Preal vs shuffled Ppaired correct-answer log p Current caveat: device microbatch = 1 rotates P mostly inside one packed sample; the strict gate must prove content sensitivity before this repair is promoted.

10 · Implementation lessons

Failure modeObserved consequenceV6 resolution / status
sample frames before building GOPstraining and inference see different motion historiesFIXED full timeline → GOP → complete-GOP selection
max_frames=16 for I+16Ponly I+15P fitsFIXED Stage 1 uses 17 sampled frames
1,377 visual-token budgetnominal video training covers too little temporal evidenceFIXED 8,192-token pilot
live cache misses in distributed trainingCPU motion search creates rank stragglersFIXED cache-only deterministic scan + independent prefill
visual-side drift in Stage 2breaks the AdaCodec freeze contractFIXED zero visual trainable parameters
ordinary answer CELLM can solve from I/question priors and suppress PFAILED GATE add explicit P-dependence objective
within-pack weak corruptioncounterfactual pair remains semantically similarFAILED GATE move corruption before microbatch split
packed sample maskspossible cross-example leakageVERIFIED packed/unpacked masks match; no cross-sample attention

11 · Release gates for the next Stage-2 run

GatePass conditionCurrent state
P-content dependencepaired correct-log-prob CI(real−zero) and CI(real−foreign) both strictly above 0FAILED AT STEP 820
short deployable Preal P16 ≥ zero-shot 58.5% without Hφ regression65.0% / 62.0%
long codec8K codec > zero-shot 72.5%, not merely recover it72.5% · NOT YET ABOVE
native retention8K dense does not regress80.5% vs 77.0%
repeatabilitystep-900 n=200 gates reproduce the directionQUEUED
Evidence snapshot: Stage-1 step17000; Stage-2 ordinary step0–800 and weak-donor counterfactual repair step820; MLVU exact n=200 unless noted. The failed strict P-content gate is explicitly labeled.