Molmo2 Codec V6 — Performance Record

Verified Stage-1 evidence and the exact step-1200 MLVU artifact · 2026-08-03

V6Stage 1Stage 2MLVUcheckpoint record

1 · Verified headline

Step-1200 real-P result: 144 / 200 = 72.0% raw MLVU option accuracy.

Item	Verified value
Stage-2 checkpoint	`codec_stage2_mixed8k_popen_step2000/step1200_consolidated.pt`
P-tokenizer checkpoint	`codec_stage1_full_v6_delta/ptokenizer_step17000.pt`
Evaluation samples	`20260802_072043_samples_molmo2_mlvu_dev.jsonl`
Model implementation	`/fsx/home/weikai.huang/molmo2_codec/mm_olmo`
Backend	`video_backend=codec`, `p_variant=real`
Visual contract	8,192 visual-token budget · 2,048-frame timeline cap · 16,384 total sequence length
Raw result	144 / 200 = 72.0%
lmms-eval aggregate	`mlvu_percetion_score=46.95141008505063`

The separate paired p_usage script reports 71.0% on a different sample set; it is not the source of the 72.0% claim.

GOP length	Real P	Zero P	Foreign/shuffled P
1	1.1719	1.3539	1.4852
4	1.1881	1.4568	1.5580
8	1.1728	1.5097	1.5823
16	1.1693	1.5429	1.5661

At GOP 16, matched P lowers held-out reconstruction loss by 24.2% vs zero P and 25.3% vs foreign P.
Late reconstruction cosine similarity is approximately 0.52–0.58: useful but not near-lossless.
This supports P-content learning, but does not by itself prove that a Stage-2 LLM uses P tokens.

Version	Downstream status
V4	Legacy Stage-2 records around 50.5–52.0%; not directly comparable without the same harness.
V6-delta step 17K	Verified in the 72.0% step-1200 real-P run.
Historical `codec_stage2_v6_step17000/step8800`	Separate recipe and evaluation lineage. Not the source of 72.0%, not current progress, and not a V6+HEX result.
V6+HEX `ptokenizer_step20000.pt`	Stage-1 checkpoint exists; no matched Stage-2 MLVU result yet.

Next fair comparison: hold the 200-example MLVU list, Molmo2 evaluator, Stage-2 recipe, and 8K contract fixed.
Swap only the P-tokenizer and matching HEX cache/gamma artifacts.
Report raw option accuracy and the official aggregate as separate columns.
Do not infer current training progress from the historical step8800 directory.

Verified artifact record · raw accuracy and lmms-eval aggregate intentionally reported separately · static report