Molmo2 Codec — P-Token & Training-Interface Audit

Primary failure

Wrong feature contract

P frames bypass Molmo's pretrained multi-layer attention-pool + MLP interface.

Same output count

16 ≠ 64→16

The final token budget matches; the internal capacity, spatial topology, and connector do not.

Repair target

Molmo-native V6

64 latent states, block mask, layers −3/−9, 2×2 attention pooling, vision-frozen Stage 2.

2 · What 64→16 means

64 intermediate latent states

→

16 final P tokens

Each color-coded 2×2 latent group becomes one final token through attention pooling; this is not simple averaging.

Each latent can still attend to the full patch sequence; the 8×8 arrangement structures the native merger input.
The intermediate representation has four times the token capacity while preserving the same 16-token LLM budget.
For Molmo, the appropriate merger is its pretrained attention_meanq + MLP, not Qwen's module copied literally.

3 · Block attention mask

AdaCodec mask

Key: patch

Key: latent

Query: patch

allow

block

Query: latent

allow

Current V5

Key: patch

Key: query

Query: patch

allow

Query: query

allow

Reference property: appended latents can aggregate all patch information without changing the pretrained patch-token computation.
Current behavior: query states can influence patch states at every transformer layer.
Required unit test: changing latent-query values must leave patch hidden states unchanged to numerical tolerance.

4 · Native-interface mismatch

Axis	Molmo I-frame path	Current P-frame path	Target P-frame path
ViT states	layers −3 and −9	final layer only	latent states from layers −3 and −9
Pooling input	9 patch states per output	none	four latent states per output
Pooling	pretrained attention_meanq	none	P-specific copy initialized from attention_meanq
Projection	pretrained Molmo MLP	fresh Linear 1152→2560	P-specific native MLP copy
Output family	81 × 2560	16 × 2560, unrelated bridge	16 × 2560, native feature contract

LLM sees

Two visual languages

I frames use the pretrained connector; P frames use a newly learned linear mapping.

Not required

Copy Qwen DeepStack

Molmo has its own base-native interface; matching that contract is the goal.

Required

Distribution parity

P-token norms, covariance, and connector outputs should resemble native visual tokens before Stage 2.

5 · The Stage-1 → Stage-2 hand-off is broken

Stage 1 trains
E_P + 16 queries + e_t + Hφ
ptok.proj excluded

→

Proxy proves
P tokens contain video-specific signal
last-layer 3×3 teacher proxy

↛

Stage 2 consumes
query → untrained Linear → LLM
E_P frozen

Finding	Current evidence	Why it matters	Priority
P projection absent from Stage-1 optimizer	`train_params` includes E_P/query/e_t/Hφ, not `ptok.proj`	The deployed P-to-LLM bridge remains at initialization	P0
Teacher target is disjoint	final ViT layer + adaptive average pool 3×3	Proxy success does not establish native-connector compatibility	P0
Stage 2 trains visual bridges	`ft_connector=True`; P projection explicitly trainable	I and P visual interfaces drift differently during short SFT	P0
Ablation defaults to n=1	training loop calls `ablation_eval()` without a chain length	Deep-GOP information is never monitored by the headline proxy	P1

6 · Corrected training contract

Stage 1
train E_P + 64 latents + P connector + e_t + Hφ

→

Shared feature contract
layers −3/−9 → attention pool → native MLP

→

Stage 2
freeze every visual module; train LLM only

Module	Stage 1	Stage 2
I-frame ViT + connector	frozen teacher	frozen
E_P ViT	train	freeze
64 latents + temporal embedding	train	freeze
P-specific native connector	train	freeze
Hφ	train	discard
LLM	freeze	train

7 · Temporal coverage and evaluation symptom

V5 fixes

Non-zero t=1…16

Fixed sinusoidal rows eliminate the V4 zero-collapse.

V5 still trains

At most 8 source frames

The launcher omits max_frames; the trainer default remains 8.

Measured result

−23 MLVU points

Dense 75.0 vs V4 codec 52.0 at 33% token budget, n=200.

AdaCodec codec − dense

+3.1

Molmo V4 codec − dense

−23.0

Bar direction is encoded by color; lengths show absolute delta on a 25-point reference scale. Cross-system absolute scores are not directly comparable.

Use max_frames=17 for one I-frame plus 16 P-frames.
Balance chain buckets rather than sampling proportional to their counts.
Report real/shuffled/zero at n={1,2,4,8,12,16}; add token norm and covariance diagnostics.

8 · Minimal V6 gate

Gate	Test	Pass criterion
A	Block-mask invariance	Patch states unchanged when latent-query values change
B	64→16 native connector smoke	16 × 2560 outputs; stable norm; I path bit-identical
C	5k Stage-1 native-space proxy	real < shuffled ≈ zero at short and deep GOP positions
D	500/1500-step LLM-only Stage 2	MLVU n=200 ≥57 at matched codec budget
Scale	130k Stage 1 + long Stage 2	Only after A–D pass

Molmo2 Codec — P-Token & Training-Interface Audit

1 · Verdict

2 · What 64→16 means

3 · Block attention mask

4 · Native-interface mismatch

5 · The Stage-1 → Stage-2 hand-off is broken

6 · Corrected training contract

7 · Temporal coverage and evaluation symptom

8 · Minimal V6 gate