Molmo2 Codec — Motion Search & Adaptive GOP Audit

Core design

Correct direction

Own motion search on decoded 2-fps frames; no dependency on playback-codec I/P/B syntax.

Implementation status

Strong but inconsistent

ViT-aligned blocks, wide search, deterministic cache — plus several verified scale and GOP bugs.

Highest-risk finding

Hidden λ rescale

The configured λ is multiplied by 14², so λ=0.05 becomes an effective displacement coefficient of 9.8.

Distinctive characteristic	Current implementation	Why it matters
MLLM-oriented, not playback-oriented	Fresh search over sampled RGB frames	Frame roles and motion units are chosen for tokenization rather than bitstream compliance
ViT-aligned coding unit	14×14 macroblocks on a 27×27 grid at 378 px	Each visual patch receives one uniform motion vector
Low-fps displacement coverage	±48 px window at 2 fps	Covers the observed p95 motion magnitude near 47 px
One search, two products	Selected MV/residual + summed pcost	The same work builds P inputs and triggers adaptive I-frame insertion
Deterministic encoding	Fixed scan order, strict tie-break, cache fingerprint	Stage-1 and Stage-2 can consume reproducible codec inputs
Compact offline cache	729 block MVs/frame; rebuild residual at 2–4 ms/frame	Avoids live block matching inside distributed training

2 · Search objective and candidate schedule

Paper Eq. 1

SAD + λ‖d‖₁

SAD is explicitly defined as a sum over the macroblock.

Current code

SAD + λ·196·‖d‖₁

pen = lam * block * block, with block=14.

Current λ

0.05 → 9.8

Effective per-pixel displacement coefficient before multiplying by |dx|+|dy|.

full search ±8

289

hierarchical ±16

≤297

current hierarchical ±48

≤2409

Candidate positions per block: step-2 coarse grid plus at most eight local-refinement positions.

Verified: hierarchical = search > 8; current ±48 is approximate coarse-to-fine search, not exact full-window search.
Verified: module comments claim an “exact minimizer of Eq. 1,” but the default wide-window execution is approximate and the configured λ is silently rescaled by block area.
Evidence boundary: the paper does not publish its numerical λ; the verified problem is parameter/documentation semantics, while the quality impact requires a λ ablation.
Not inherently wrong: AdaCodec itself uses approximate hexagonal search with local refinement; the issue is objective scaling and inaccurate documentation.

3 · Search-range evolution and measured tail behavior

±8 boundary hits, p90 videos

15.6%

±8 boundary hits, worst video

55.9%

±16 clipped blocks at 2 fps

24.5%

±48 clipped blocks at 2 fps

≈5%

Version	Sampling / range	Characteristic
V1	native-rate consecutive frames, ±8	Small motion but high-motion tail repeatedly hits the boundary
V2	native-rate, ±16 hierarchical	Twice the reach at approximately full-±8 candidate cost
V3–V5	2 fps, ±48 hierarchical	Correct temporal spacing and broad reach; approximately 8× more candidates than V2

Local measurements are historical diagnostics from the repository documentation, not a standardized benchmark dataset.

4 · The motion-channel saturation bug

Raw motion	Current encoded value, scale 16	Proposed value, scale 48
8 px	0.50	0.17
16 px	1.00	0.33
32 px	1.00	0.67
48 px	1.00	1.00

Verified: current input uses clip(motion / 16, −1, 1) despite a ±48 search window.
Verified: Stage-1 cache construction calls CodecFrame.u5() without passing cfg.mv_scale; changing the config field alone does not change this path.
Required fix: pass the configured scale explicitly and normalize by search_range; regenerate caches under a new fingerprint.

5 · Adaptive GOP: good idea, three concrete bugs

Intended Kmax=16

I12…1516I

Current condition

I12…15I

The code increments since_i and then checks >= 16, so frame 16 becomes I; the maximum P-chain length is 15.

Finding	Evidence	Impact	Priority
Per-video γ	`gamma=None` invokes quantile calibration on each video	Every video is pushed toward a similar GOP distribution; content adaptivity is weakened	P1
Global calibration at wrong fps	`frame_costs()` decodes without `sample_fps=2`	The available global-γ helper calibrates native-rate motion but training/eval use 2-fps gaps	P0
Kmax off by one	`since_i += 1; if since_i >= max_gop`	Declared Kmax=16 produces at most 15 P-frames	P1
`gop_min_p` unused	Config field never enters `assign_roles()`	Configuration and behavior can silently diverge	P2

6 · Strengths worth preserving

Verified quality

Residual energy drops

Historical tests report −84.7% vs. naive difference and +23%/+51% gains on selected high-motion clips.

Training reliability

Offline deterministic cache

Codec work is removed from the distributed hot path; a config fingerprint prevents most stale-cache reuse.

Base alignment

14×14 motion cells

The motion field respects Molmo's ViT patch grid rather than playback-codec dynamic partitions.

Keep the own-search design, previous-sampled-frame reference, signed residual, and five-channel representation.
Keep the wide low-fps window, but revalidate ±32/±48 after correcting the objective scale.
Keep compact block-MV caching and fast runtime residual reconstruction.
Update the stale codec_preprocess.py module docstring, which still says motion comes from the bitstream.

7 · Repair and validation gates

Gate	Change	Measurement	Pass condition
A	Match Eq. 1 exactly or document a normalized-SAD equivalent	Exact-search oracle on a sampled block set	Hierarchical selected cost within a predefined tolerance; no hidden 196× λ
B	Calibrate one γ on 2-fps training frames	GOP-length histogram by dataset and motion quartile	Median near 8; dynamic videos receive shorter GOPs than static videos
C	Fix Kmax and motion normalization	Chain lengths and encoded-MV histograms	t=16 occurs; saturation limited to true ±48 boundary
D	Range/λ ablation: ±32 vs ±48; λ sweep	residual MAE, pcost, runtime, boundary rate	Choose the Pareto point before rebuilding the full cache
E	End-to-end matched-token evaluation	MLVU/MVBench plus quality by GOP depth	Search fixes improve P quality without collapsing temporal coverage