Goal. Port AdaCodec (a predictive video codec for MLLMs — I-frames keep full tokens, P-frames are compressed to a handful) onto Molmo2-4B-SFT, so long video costs ~4× fewer visual tokens at the same temporal coverage.
Where we are. A long diagnosis found the real blocker was not frame sampling or the loss — it was that our P-tokenizer was built ~1000× too small: we froze the ViT and trained only ~0.49M query params, while AdaCodec trains a separate ~576M ViT branch. After rebuilding it as a trainable ViT branch (V4), a held-out proxy shows the compressed P-tokens finally carry video-specific content that the frozen version could not. Stage-2 accuracy confirmation is in progress.
MLVU multiple-choice, 32 frames, n=100 (dense baseline via the official eval path).
| Config | P-tokenizer trainable | MLVU acc | visual-token budget | Stage-1 proxy (real<shuf) |
|---|---|---|---|---|
| Dense baseline (Molmo2-4B-SFT, per-frame RGB) | — | 75.0% | 100% | — |
| V2 codec (N_P=8, 30fps, search ±16, frozen ViT) | 0.49M | 35.0% | 25.3% (~4×) | ~0.007 |
| V3 codec (N_P=16, 2fps, ±48, frozen ViT) | 0.49M | not run (proxy weak) | ~25% | ~0.007 (flat) |
| V4 codec (+ TRAINABLE E_P branch) | 381M | in training | ~25% | ~0.10 (grew ~14×) |
Read: V2 codec at ~4× compression scored 35% vs dense 75%. V3's frame/search fixes did not move the Stage-1 proxy. V4's trainable P-tokenizer branch is the first change that moved it.
The Stage-1 proxy trains a co-trained readout to predict the target frame's features, then feeds it (a) the correct video's P-tokens, (b) shuffled P-tokens from a wrong video, (c) zeroed P-tokens — all on held-out videos. If P-tokens carry video-specific info, correct << shuffled ≈ zero.
| realP (correct) | shufP (wrong video) | zeroP (none) | meaning | |
|---|---|---|---|---|
| V3 (frozen 0.49M) | 0.46 | 0.47 | 0.72 | shuf ≈ real → generic bias, not video-specific |
| V4 (trainable 381M) | 0.21 | 0.30 | 0.29 | shuf ≈ zero, real wins → video-specific |
V3's P-tokens fit any video (a wrong video's tokens helped as much as the right one's). V4's trainable branch encodes what is specific to this video — a wrong video's tokens are now worthless (≈ none).
Note: the absolute values above are at different readout-training points, so compare the relative pattern (shuf vs real vs zero), which is robust. A same-readout-steps fair comparison is running and will replace the absolutes.
The size gap (root cause):
| P-frame ViT | trainable? | P-tokenizer trainable params | |
|---|---|---|---|
| AdaCodec (paper Appx E) | ~576M (Qwen3-VL) | yes (E_P branch) | ~576M |
| Ours V1–V3 | 381M (SigLIP2) | frozen, shared | 0.49M |
| Ours V4 | 381M (copy) | yes (E_P branch) | ~381M |
Held-out video-specific signal (real<shuf) over V4 Stage-1 steps — grew ~14× vs the flat V3 value, and the pattern flipped from "generic" to "video-specific". Still stable (no overfitting on held-out).
| V4 step | 200 | 600 | 1000 | 1400 | 6000 | 7800 | 8400 |
|---|---|---|---|---|---|---|---|
| real<shuf | 0.006 | 0.055 | 0.096 | 0.100 | 0.059 | 0.100 | 0.080 |
(V3, frozen: ~0.007 flat across all steps and chain lengths — the proxy never moved.)
Full audit vs the AdaCodec paper (their codec code is not open-sourced; implemented from the paper).
| Axis | AdaCodec | Ours (status) |
|---|---|---|
| P-tokenizer | trainable ~576M ViT branch | V4: trainable 381M branch ✓ (was frozen 0.49M — the root cause) |
| Frame sampling | 2 fps | V3: 2 fps ✓ (was 30fps-consecutive; residual −42%) |
| Motion search window | enlarged for low-fps | V3: ±48 ✓ (was ±16, clipped 24.5% of blocks at 2fps) |
| N_P per P-frame | 16 | 16 ✓ (was 8) |
| Stage-1 LR | 1e-4, warmup+cosine | V4: 1e-4 + 4k warmup ✓ (was 1e-3 flat) |
| Objective | ‖·‖₁ + (1−cos) feature-match | same ✓ |
| u5 / widened embed / residual / GOP / e_t | — | aligned ✓ |
| block attention mask | patch ⊬ query | not yet (secondary; E_P trainable can adapt) |
| N_I (I-frame tokens) | 256 | 81 — base-dictated (SigLIP 3×3 pool) |
| Base model | Qwen3-VL-8B | Molmo2-4B — base-dictated |
Remaining gaps (N_I=81, 4B base) are dictated by the Molmo2 backbone and are hard ceilings vs AdaCodec's numbers even after full alignment.