← Back to project
Molmo2 Codec V6 + HEX — Step-100 Evaluation Record
Seven paired n=200 benchmarks · 8K visual-token cap · Dense versus Codec · 2026-08-06
V6HEXStage 1Stage 2paired evaluation
1 · Result
Codec-format performance: the step-100 model reaches 81.0% on the fixed MLVU n=200 set, versus 76.5% through the original Dense input path.
| Input path | Correct | Accuracy | Delta |
|---|
| Original Dense RGB | 153 / 200 | 76.5% | — |
| Our Codec | 162 / 200 | 81.0% | +4.5 |
- Both paths use the same trained model and the same fixed 200 questions.
- Codec represents an average 121.3 timestamps under 8K, versus 101 Dense frames, giving denser temporal coverage.
- The expanded six-benchmark suite below tests whether this advantage generalizes beyond MLVU.
The fixed MLVU comparison changes only the visual representation: original Dense RGB versus our Codec path.2 · Checkpoint lineage
| Artifact | Exact identity |
|---|
| Molmo base | Molmo2-4B-SFT / step30000 |
| Stage-1 source | codec_stage1_full_v6_hex_windows2/ptokenizer_step20000.pt |
| Stage-1 contract | V6 · HEX motion · raw paper Hφ · learned queries · residual-native content auxiliary · no anti-shortcut loss |
| Stage-2 milestone | codec_stage2_8k_hex_validation/step100_consolidated.pt |
| Stage-2 trainable components | LLM · P-ViT · P-connector; native I-ViT and connector frozen |
| Evaluation | MLVU MC · fixed stratified n=200 · visual budget 8,192 |
- The evaluator loads the full standard Stage-2 model state and 421 P-tokenizer tensors; 421 is not the total number of loaded tensors.
- The current anti-shortcut Stage-1 run is a separate lineage and is not responsible for the 80–81% result on this page.
3 · Visual path
- I frames remain native Molmo visual tokens: RGB → frozen ViT → frozen connector → 81 tokens.
- P frames use motion plus warped-reference residual: P-ViT → learned latents → independent connector → 16 tokens.
- Both branches share the same timestamp, position, visual-token, and attention-mask interface at the LLM boundary.
- HEX changes motion search; it does not by itself align P-token features to the Molmo distribution.
The native I path is preserved; only the predictive P branch is learned.4 · GOP selection and real-video audit
| Selection field | Value |
|---|
| Timeline | 2 FPS · up to 2,048 source frames |
| Adaptive refresh | new I when predictive cost crosses global γ |
| Safety refresh | no more than 16 P frames after an I |
| Budget selection | uniformly spaced complete GOPs; no orphan P frames |
| 8K MLVU trace | Codec averages 121.3 represented timestamps versus 101 Dense frames |
Predictive-cost refresh at a real composition change.
Predictive-cost refresh at an exterior/car scene cut.
Forced refresh at the 16P safety boundary.
- The former 12-card gallery was a qualitative audit subset, not training data or a benchmark result.
- Three representative clips are sufficient here; the larger GOP audit remains available as a separate report.
Two real timelines show cost-triggered and length-triggered I-frame refreshes without a product-style video gallery.5 · Stage-1 evidence
| Stage-1 evidence | Interpretation |
|---|
| Historical V6-delta step17K: matched P reconstruction loss ≈1.17–1.19 | Hφ reads video-specific P information, but reconstruction is not near-lossless. |
| V6+HEX step20K held-out reconstruction | The learned predictor reads P information and reconstructs target visual features at a useful but not near-lossless level. |
- Hφ can decode P while the LLM still ignores direct P-token inputs; these are different interfaces.
- The current public result therefore focuses on the end-to-end Dense-versus-Codec comparison.
6 · Stage-2 recipe
| Field | Current formal value |
|---|
| Sequence length | 32,768 |
| Codec visual allowance | up to 16K tokens in training; 8K in this evaluation |
| Global batch | 128 dynamically packed bins |
| Mixture | 40% long codec caption · 10% normal codec video · 37.5% native video · 12.5% replay image |
| Optimization | BF16 AMP · FSDP full shard · activation checkpointing |
| Learning rates | LLM 1e-5 · P-ViT 1e-6 · P-connector 5e-6 |
- Native vision remains frozen to preserve the original image/video interface.
- Step100 updates both the LLM and P branch; its one-point aggregate gain cannot be assigned to P-ViT alone.
7 · Fixed n=200 evaluation suite
| Benchmark | Sample | Dense | Codec | Status |
|---|
| MLVU | 200 fixed | 76.5% | 81.0% | Complete |
| Video-MME | 200 fixed | 70.0% | 69.0% | −1.0 |
| LVBench | 200 fixed | 48.0% | 49.5% | +1.5 |
| MVBench | 200 total, 10 per each of 20 tasks | 75.5% | 68.5% | −7.0 |
| TempCompass MC | 200 fixed | 69.0% | 68.5% | −0.5 |
| PerceptionTest MC | 200 fixed | 82.0% | 83.0% | +1.0 |
| NExT-QA MC | 200 fixed | 84.0% | 83.5% | −0.5 |
| LongVideoBench | 200 fixed | — | — | Awaiting gated access |
- Each benchmark uses one immutable document-ID manifest shared by Dense and Codec.
- Evaluation contract: 8,192 visual tokens, 16,384 total sequence length, full-video uniform candidate coverage, and the same prompt and parser.
- Across the five new benchmarks other than MVBench, the macro average is 70.6 Dense versus 70.7 Codec.
- MVBench is the only clear regression and localizes the next investigation to fine-grained motion and state tracking.
Our Codec matches the original Dense path across five of six new benchmarks; MVBench is the single large regression.