Molmo2 Codec — AdaCodec vs. CoPE vs. LLaVA-OV2

TL;DR

CoPE’s documented default is a 240-frame raw GOP at 30 FPS (1 I + 239 P), but fusion of 30 P frames makes the LLM stream effectively 1 I + 7 Δ groups per 8 seconds.
Our AdaCodec 2 FPS, max_p=16 spans about 8.5 seconds per GOP, so its temporal scale is already close to CoPE; the representation and budget mechanism are different.
For AdaCodec, the primary limit remains visual-token cost; over-budget selection must remove complete GOPs, not individual P tokens.

1 · Raw GOP vs. LLM-visible timeline

CoPE: re-encodes MPEG-4 at 30 FPS, GOP=240, and uses fusion s=30, yielding an effective 1 FPS stream.

AdaCodec: searches the 2 FPS timeline and emits one I/P representation per selected sampled frame; max_p=16 gives a roughly 8.5-second GOP.

The apparent “1+239” CoPE GOP is therefore not 240 independent P-token groups in the LLM.

The raw CoPE codec GOP is long, but P-frame fusion reduces the LLM-visible temporal sequence; AdaCodec reaches a similar temporal span with a shorter sampled GOP.

2 · What CoPE found

CoPE reports 1 keyframe/GOP as the strongest-compression setting with good accuracy.

4 keyframes/GOP often gives the best accuracy on PerceptionTest and NextQA, at a higher token cost.

Its Δ-Encoder emits 8 tokens per P group in the main setup; increasing to 16 showed diminishing returns.

CoPE therefore exposes a density/quality Pareto frontier rather than one universally optimal GOP.

Approximate costs use Molmo-style I=81 and our AdaCodec P=16 only to make the scale intuitive; CoPE’s actual backbone has different dense-token width.

3 · Budget policy: what we should implement

Timeline: process the complete 2 FPS video up to 2048 frames.

GOP: keep AdaCodec’s adaptive cost and max_p=16; do not replace it with CoPE’s raw 240-frame GOP.

Selection: calculate each complete GOP’s visual-token cost, retain transition/high-cost GOPs plus uniform temporal coverage, and drop only whole GOPs when the 16K visual budget is exceeded.

Training/evaluation: report coverage, I/P ratio, selected GOP count, and actual visual tokens together; never report frame count alone.

The comparable design decision is the budget policy, not the raw integer used as GOP length.

4 · Coverage and selection mechanics

One AdaCodec GOP spans approximately the same 8-second temporal scale as the documented CoPE GOP.

The 2048-frame cap is a full-timeline processing limit (1024 seconds at 2 FPS), not a claim that all 2048 frames fit into one LLM sample.

When the 16K visual budget binds, complete GOPs are selected across the timeline to preserve coverage.

The per-GOP temporal scale is short; the timeline cap controls how much video is searched, while the token cap controls what enters the LLM.

The safe AdaCodec selection path preserves predictive dependencies by dropping only complete GOPs.

5 · Final configuration table

Dimension	CoPE	LLaVA-OV2	AdaCodec (ours)
Raw sampling	30 FPS re-encode	Stage-dependent source-frame schedule	2 FPS timeline
Raw GOP	240 frames = 8 s	Codec-stream schedule; not an AdaCodec GOP definition	1 I + up to 16 P ≈ 8.5 s
LLM-visible P unit	30-frame fusion → 1 Δ group	Codec-specific visual stream	One P representation per sampled frame
P representation	8 Δ-tokens (main setup)	Not directly comparable	16 P-tokens (current checkpoint)
Primary budget knob	Keyframe density + P fusion	Source-frame budget	Visual-token cost + adaptive GOP
Over-budget action	Reduce density / fuse P groups	Sampling schedule	Drop complete GOPs only
Our formal setting	Reference only	Reference only	`2 FPS, max 2048 frames, max_p=16, 16K visual / 32K seq`

Dimension

CoPE

LLaVA-OV2

AdaCodec (ours)

Raw sampling

30 FPS re-encode

Stage-dependent source-frame schedule

2 FPS timeline

Raw GOP

240 frames = 8 s

Codec-stream schedule; not an AdaCodec GOP definition

1 I + up to 16 P ≈ 8.5 s

LLM-visible P unit

30-frame fusion → 1 Δ group

Codec-specific visual stream

One P representation per sampled frame

P representation

8 Δ-tokens (main setup)

Not directly comparable

16 P-tokens (current checkpoint)

Primary budget knob

Keyframe density + P fusion

Source-frame budget

Visual-token cost + adaptive GOP

Over-budget action

Reduce density / fuse P groups

Sampling schedule

Drop complete GOPs only

Our formal setting

Reference only

2 FPS, max 2048 frames, max_p=16, 16K visual / 32K seq

6 · Open A/Bs

CoPE-style P fusion at 2 FPS: fuse 2 consecutive P primitives into one temporal Δ group.

AdaCodec P-token count: 8 vs. 16.

Selection policy: cost-only vs. cost plus uniform full-video coverage.

Sources: CoPE-VideoLM paper; CoPE project page. AdaCodec configuration is the current Molmo2 Codec design record.