← Back to project

Molmo2 Codec V6 · Two Benchmark Groups

Current verified scorecard · 2026-08-08

V6evaluationLLaVA-OV2long video

1 · Letian / LLaVA-OV2 benchmark group

Current model: V6+HEX Stage-1 step20K + Stage-2 step500.
Purpose: compare our 4B model with released LLaVA-OV2 8B.
Video rows use the exact same frozen 200 questions and source-frame protocol.

Benchmark	Released LLaVA-OV2 8B	Molmo2 Codec 4B	Status / protocol
MMBench	85.82	84.11	Full split · native image path
MMStar	64.36	63.18	Full split · native image path
OCRBench	77.20	70.30	Full split · native image path
ChartQA	85.80	80.24	Full split · native image path
Video-MME Short	81.0%	Dense 77.0% · Codec 78.0%	Exact same 200 · 1 FPS · 128 source frames
Video-MME Medium	72.5%	Dense 67.0% · Codec 66.0%	Exact same 200 · 1 FPS · 256 source frames
MLVU	81.5%	Dense 69.93% · Codec 72.05%	Exact same 200 · 1 FPS · 512 source frames

2 · Long-video paired benchmark group

Current complete record uses Stage-2 step500.
Purpose: test Dense-to-Codec stability across six video tasks.
Each row uses an exact paired 200-sample manifest, an 8,192-token visual budget, 2 FPS full-span sampling, and a 16,384-token sequence length.

Benchmark	Dense	Codec 8K	Codec − Dense
Video-MME	68.5%	68.5%	0.0
LVBench	46.5%	49.0%	+2.5
MVBench	76.5%	71.0%	−5.5
TempCompass	69.5%	69.0%	−0.5
PerceptionTest	83.0%	83.5%	+0.5
NExT-QA	84.0%	83.5%	−0.5
Unweighted mean	71.33%	70.75%	−0.58

3 · Status and comparability

Item	Checkpoint	State	Next action
Letian / OV2 benchmark group	Stage-2 step500	Complete	Use as the released-model comparison snapshot
Long-video paired group	Stage-2 step500	Six tasks complete; exact paired IDs verified	Use as the Dense-to-Codec stability snapshot

Both groups now use the same Stage-2 checkpoint, but their task sets and video sampling contracts differ, so their means are reported separately.
Video-MME appears in both groups but uses different frozen evaluation contracts.

Verified from local lmms-eval result artifacts. Every reported video row is complete; no provisional scores are included.