Molmo2 Codec V6 — Architecture, Evidence & Training Readiness

Architecture

Core V6 is aligned

Molmo-native connector, block mask, 144→16 topology, temporal code, and Stage-2 visual freeze are implemented.

Motion search

Keep current search

Real-video oracle audit shows near-exact cost/ranking at 3.66× the speed of exhaustive search.

Training status

Alignment gate is open

The runtime is now viable, but RMS-only alignment does not yet beat shuffled P tokens and tuple-GBS 128 still needs accumulation.

Decision: freeze the V6 architecture and Nₚ=16; repair the training/data path before rebuilding caches or launching formal training.

Current target remains valid: I + P₁…Pₙ predicts the teacher representation of RGB(Pₙ); the paper wording is ambiguous, but the deployed representation makes this the coherent objective.
Do not interpret passing component tests as end-to-end readiness: the remaining risks live in batching, sampling, resume, cache, and evaluation contracts.
Evidence labels: verified behavior, paper mismatch, and unpublished implementation choices are kept separate throughout this report.

2 · V6 architecture

3 · Initialization and freezing

Contract	V6 choice	Status
P-token budget	16 final tokens per P-frame	LOCKED
Internal topology	144 learned latents as 12×12; native 3×3 connector produces 4×4	VERIFIED
Feature interface	Molmo selected layers and copied attention-pool/projector	VERIFIED
Attention mask	Patch states cannot read latents; latents can read patches	VERIFIED
Temporal range	Post-connector sinusoidal code for t=1…16 plus learned residual	VERIFIED

Component

Initialization

Stage 1

Stage 2

I-frame ViT + connector

pretrained Molmo

frozen

Motion search / block matching

fixed deterministic algorithm

non-learned

P patch stem: residual/RGB columns

exact pretrained-stem copy

train

frozen

P patch stem: MV columns

zero init, paper-aligned

train

frozen

P ViT

pretrained copy

train

frozen

P connector/projector

pretrained copy, independent storage

train

frozen

144 latent queries

non-zero learned init

train

frozen

t_scale / t_delta

8 / zero

train

frozen

Hφ predictor

fresh transformer

train

discard

LLM

Molmo2-4B-SFT

frozen

train

Terminology: “MV columns” are the learned two-channel patch projection; codec motion search itself has no trainable parameters.
Numerical parity: the widened stem matches the original RGB stem exactly; copied P/I connector and projector outputs match at initialization.
Paper-aligned zero init: all 64/64 MV test weights receive a non-zero first-step gradient (norm 2.89e-5).
Checkpoint guard: Stage 2 now rejects a checkpoint if the legacy motion sidecar diverges from the same weight stored inside ptok.
Stage-2 correction: the I-side connector is now frozen together with the rest of the visual stack.

4 · Verification evidence

Regression suite

34 passed

V6 A/B, P-tokenizer, vision-only teacher, bucket semantics, and synchronized-DDP sampling tests.

Initialization parity

max diff 0

RGB stem, initial connector output, and initial projector output.

Gradient reachability

all live

motion, query, pooling, projector, t_scale, and observed t_delta rows.

Check	Measured result	Interpretation
Temporal rows t=1…16	norm ≈ 1; minimum pair distance 0.445	No V5 zero-collapse
Query initialization	norm 0.6601	Latents are not dead at initialization
Zero-initialized MV projection	64 / 64 gradient entries non-zero; norm 2.89e-5	Zero init preserves the pretrained starting point without blocking learning
Duplicated motion checkpoint entries	strict equality required	Stage 2 fails fast instead of loading a split-brain motion projection
Visual parameters trainable in Stage 2	0	Paper freeze contract is now met
LLM parameters trainable in Stage 2	non-zero count	Freeze does not disable language fine-tuning
Current implementation commit	5e6cb1d7	Includes Stage-2 connector freeze and motion-checkpoint consistency guard

These checks validate component wiring and gradient flow; they do not validate the distributed objective, dataset split, cache distribution, or resume semantics.

5 · Motion-search oracle audit

MV agreement

94.42%

Hierarchical search vs. exhaustive ±48 oracle on 66 real 2-fps frame pairs.

Frame-cost rank

ρ = 0.99967

Adaptive-GOP ordering is effectively preserved on this sample.

Speed

3.66× faster

0.355 s vs. 1.300 s per frame pair at production geometry.

MV exact agreement

94.42%

frame pcost mean regret

0.593%

frame pcost p95 regret

2.093%

prediction MAE / 255

0.217

Regret bars are scaled to a 2.5% visual range; MAE is in 8-bit pixel levels.

Decision check	Result
High-cost selections at 10 / 20 / 30 / 50%	identical
Recalibrated γ difference	0.0019%
I/P role agreement and I-boundary Jaccard	100% / 100%
Pathological random-texture stress test	fails badly; retained as an explicit adversarial limit

Decision: retain rgb-valid-hier-v1; exhaustive search is the stronger oracle and is too slow for the full pipeline.
Before cache freeze: repeat on 1,000 actual training videos with 2–4 random pairs/video.
Pass gates: frame-cost p95 error <3%, rank correlation >0.995, role agreement >98%, boundary Jaccard >0.95, prediction MAE <1/255.

6 · Remaining training gates

Priority	Required change	Required regression test
P0	Use t=1…16, random video windows, and true sample-weighted global batch 128	Multi-rank objective matches a single-process reference
P0	Fail collectively on empty/error ranks; stop swallowing configuration errors	Injected rank-local failure exits every rank with the same root cause
P0	Build deterministic video-level proxy splits; run paired controls in eval mode	No video identity overlap; repeated evaluation is deterministic
P0	Make resume atomic and distinguish warm start from exact resume	Interruption, digest mismatch, and off-by-one resume tests
P0	Implement full 2-fps timeline → adaptive GOP → token-budget selection	Long-video fixture constructs GOPs before budgeted sampling
P1	Unify Stage-1/Stage-2 signed-u5 transport and repair eval double processing	Stage-1, Stage-2, and eval connector outputs match
P1	Replace cache forward-scan with an eligible-index manifest	Eligible examples are sampled uniformly; code errors are not hidden

7 · Fixed choices vs. open experiments

Item	Decision	Reason
Nₚ	16	Matches AdaCodec/JoyAI's approximate per-P budget and preserves the current Molmo adaptation
Motion search	keep hierarchical	Near-exact real-video behavior at 3.66× lower runtime
Stage-1 target	RGB(Pₙ)	Consistent with what the deployed P representation actually encodes
Temporal range	1…16	Matches Kmax=16 and is numerically non-degenerate
Motion oracle expansion	required	Current 66-pair audit is strong but not sampled from the final training distribution
Signed-u5 quantization	A/B	Current uint8 path introduces a known zero offset and train/deploy shift
Molmo 81:16 budget ratio	measure	Absolute Nₚ matches the reference; total-token compression is lower than Qwen's 256:16 ratio

Architecture choices are now frozen to prevent moving the cache contract during the readiness repair.
Open experiments are bounded by explicit metrics; none requires an untracked architecture rewrite.

8 · Formal-training data scale

Selected physical corpus

771.7k videos

All 766,738 non-Ego4D files plus 5,000 task-balanced Ego4D clips; 3.809 TB on disk.

Molmo2 reference

2.4M Video-QA examples

30k × 128 = 3.84M SFT draws, followed by 2k × 128 = 256k long-context draws.

AdaCodec reference

3.904M examples

45k × 128 = 5.76M Stage-2 draws; Stage 1 uses 130k × 128 = 16.64M tuples.

selected unique videos

0.772M

Molmo2 Video-QA pool

2.4M

AdaCodec instruction pool

3.904M

Quantity	Formal V6 target	Why
Unique video files	771,738	Keep every non-Ego4D file plus 5,000 Ego4D clips balanced across four QA styles
Stage-1 training	16.64M tuples	Match AdaCodec's 130k steps at global batch 128; draw random windows and GOP depths from the video pool
Stage-2 instruction pool	2.4M–3.9M examples	Molmo2 is the lower reference; AdaCodec is the reproduction target
Stage-2 optimization	5.76M draws	Match 45k steps at global batch 128, including 40k at 64k and 5k at 224k visual tokens

Full-source comparison: the original 820,108-file pool was 10.53 TB; QA-aware Ego4D sampling reduces the selected corpus to 3.809 TB while retaining approximately 98% of the rounded 2.4M Video-QA pool.
Do not count files as examples: one physical video can support multiple instructions, GOP selections, and Stage-1 windows.
JoyAI-VL-Interaction: its report does not disclose a corpus count and explicitly describes the released data mixture as early and not yet scaled; it is not a numerical target.
Migration rule: copy raw videos and metadata, then rebuild V6 caches after the motion, gamma, and sampling contracts are frozen.

9 · Stage-1 H200 throughput and compile decision

BF16 + DDP baseline

0.199 s / step

642 samples/s at n=1, 16 samples/GPU × 8 GPUs; 19.72% dense-BF16 MFU.

default fullgraph compile

0.171 s / step

748 samples/s; +16.4% throughput and approximately 22.99% dense-BF16 MFU.

real bucketed pipeline

+5–12%

End-to-end gain is smaller because the frozen Molmo teacher remains eager.

manual + FP32

126.7/s

DDP + FP32

128.9/s

manual + BF16

592.8/s

DDP + BF16

642.2/s

DDP + BF16 + compile

747.6/s

The first four bars use 642.2/s as the visual reference; compile exceeds that reference by 16.4%.

System choice	Measured result	Decision
BF16 Tensor Cores	approximately 5× faster than FP32	DEFAULT
DDP gradient buckets	approximately +8% over manual all-reduce in the synthetic core	DEFAULT
`torch.compile(default, fullgraph=True)`	0.199 → 0.171 s/step; first graph approximately 55 s	DEFAULT
`reduce-overhead`	0.170–0.171 s/step plus an extra approximately 14 s capture	REJECT
`max-autotune`	>7 min of duplicated rank-local search; most GEMMs still selected cuBLAS	REJECT
compile frozen teacher	fullgraph fails at data-dependent `aten.nonzero`	KEEP EAGER

Finite graph set: real Stage 1 has at most seven static chain shapes; first use costs 25–55 s, and repeated shapes return to 0.2–0.3 s end-to-end.
DDP sampling repair: bucket lengths use the cross-rank intersection, counts are all-reduced once, and every rank samples the same chain length while drawing independent examples.
Long-chain core: n=7 measures 32.75 TFLOPs/GPU/step; eager 0.174 s = 19.0% MFU and compile 0.150 s ≈ 22.1% MFU.
Batch caveat: the n=1 benchmark is true GBS 128. The current frame-budgeted real loader uses global tuple batch 8 × floor(16/n); formal alignment must accumulate microsteps to 128 tuples/update.

10 · Alignment-scale evidence and planned runs

Raw connector scale

std 664–686

Teacher maxima reach 27k–28k; the initial P output has std 0.555 and the old post-norm Hφ output was approximately unit scale.

RMS optimization

2.1187 → 1.0796

Stable 500-step optimization on 800 real videos, but optimization alone is not semantic validation.

Content-specific gate

real ≈ shuffled

RMS-only currently uses the presence/statistics of P tokens, not demonstrably the matched motion content.

Held-out proxy	real P	shuffled P	zero P	Verdict
RMS · step 200	1.2147	1.2145	1.3255	no matched-content gain
RMS · step 400	1.1477	1.1483	1.2531	+0.0006 only; below gate

Run tier	Data / optimization scale	Variants	Promotion requirement
A · loss screen	800 videos · 2k updates · 1 seed · effective 128 tuples/update	raw; RMS-only; hybrid λscale = 0.03 / 0.1 / 0.3	finite training; real P beats both controls by ≥1%
B · stability	10k videos · 5k updates · 3 seeds	top two from A; optional λP-interface = 0.01 / 0.03 / 0.1	positive gap on every seed and major source family; no scale drift
C · pilot	50k videos · 20k updates · 2k-video held-out set	single selected objective	proxy gap persists; downstream frozen-LLM activation statistics remain in-family
D · formal	130k updates · effective GBS 128 · full selected corpus	locked objective only	all contracts fingerprinted; formal γ and cache ready

Primary hybrid: normalized L1 + cosine for content, plus a log-RMS prediction term that restores magnitude information without letting 28k outliers dominate.
Interface-scale term: applied directly to P tokens, because Hφ is discarded and can otherwise absorb all scale correction; its coefficient is introduced only after the content sweep.
Non-negotiable gate: decreasing training loss or beating zero P is insufficient; matched real P must beat shuffled P on held-out videos.
Molmo contract: RMSNorm exists on the attention branch, but the raw visual embedding remains in the residual stream; pure RMS alignment is therefore an ablation, not the default endpoint.

Implementation evidence: V6 commits 2ce34fd5, 21204e8f, and 5e6cb1d7. Reference: AdaCodec and JoyAI-VL-Interaction, plus Molmo2. The public JoyAI codec implementation was not available at audit time; unpublished details are not treated as confirmed bugs.

Molmo2 Codec V6 — Architecture, Evidence & Training Readiness

1 · Executive verdict

2 · V6 architecture

3 · Initialization and freezing

4 · Verification evidence

5 · Motion-search oracle audit

6 · Remaining training gates

7 · Fixed choices vs. open experiments

8 · Formal-training data scale

9 · Stage-1 H200 throughput and compile decision

10 · Alignment-scale evidence and planned runs