[-3,-9] → attention_meanq → MLP connector.| Evidence | What it supports | Boundary |
|---|---|---|
| AdaCodec paper | Codec architecture, attention mask, 64→16 native merger, two-stage recipe, scale, benchmark and efficiency numbers | Primary reference for the codec |
| JoyAI-VL-Interaction paper | Interaction system, Qwen3-VL base, per-second decisions, 4M+ aligned clips, AdaCodec integration claim | System-level evidence, not released codec code |
| Official JoyAI repository | Released inference/system code | Codec version is still an unchecked TODO; a literal code-to-code codec comparison is impossible |
| Molmo2 codec repository | Our implementation and integration path | Audit snapshot: local commit 59320178; V5 commit is not yet on the public remote |
| Component | AdaCodec / JoyAI reference | Molmo2 V5 | Risk | Expected consequence |
|---|---|---|---|---|
| Base visual interface | Reuse Qwen3-VL native merger and matching intermediate features | Bespoke final-layer Linear for P frames | P0 | LLM receives an out-of-distribution visual embedding family |
| Latent topology | 64 latents on 8×8 grid → 2×2 merger → 16 | 16 unconstrained queries directly | P0 | Less capacity and no native spatial aggregation |
| Attention mask | Patch tokens cannot attend to latent tokens | Full self-attention | P0 | Pretrained patch computation is no longer preserved |
| Stage-1 target | Feature prediction aligned to the deployed visual stack | Final ViT layer → adaptive 3×3 average pool → Hφ | P0 | Proxy success does not establish LLM-interface compatibility |
| Stage-2 freeze | Discard Hφ; freeze visual-side modules; update LLM | Freeze E_P; train P Linear and Molmo connector with LLM | P0 | Short SFT must repair the interface while the two visual paths drift differently |
| Temporal coverage | e_t trained over GOP positions | Sinusoidal e_t covers 1…16, but Stage-1 launcher still uses 8 frames | P1 | No zero rows, but no learned deep-GOP behavior |
| Motion values | Motion passed to widened stem without the documented ±16 clamp | ±48 search; divide by 16; clip to ±1 | P1 | All motion components above 16 px collapse to the same magnitude |
| GOP threshold | One globally calibrated γ | gamma=None → per-video calibration | P1 | Each video is forced toward a similar I/P ratio; token allocation is less content-adaptive |
| New-host portability | Environment-specific, but launch inputs must resolve | V5 launcher and Stage-1 trainer still hardcode the old /weka/... project, model, data, cache, and Conda paths | P1 | V5 cannot launch on the current host without path parameterization |
| P-tokenizer fallback safety | Registered E_P or a safe explicit fallback | self.vit_p = None sits after a return and is unreachable | P2 | Current attach paths mask it; an unattached fallback can raise AttributeError |
| 2 fps, patch-aligned blocks, trainable E_P, N_P=16 | Required | Implemented | Aligned | These are not the remaining blocker |
| Molmo DeepStack / 3D M-RoPE | Qwen-specific integration | Molmo base has neither | P2 | Do not copy Qwen mechanics literally; preserve Molmo's own connector contract instead |
ptok.proj is saved in Stage 1 but excluded from the optimizer; Stage 2 loads and trains it.| Axis | AdaCodec reference | Our observed / planned state | Read |
|---|---|---|---|
| Stage 1 | 130k steps, GBS 128, LR 1e-4, 4k warmup, cosine | V4 checkpoint 60k; V5 plans 130k, BS 16 per process on 8 GPUs | V5 step count matches on paper; architecture does not |
| Stage 2 | 45k steps, GBS 128; 40k at 64k token budget + 5k at 224k | 1500-step matched result; 3000-step trial; GBS 64; 8k context | Large scale gap, but scaling a mismatched interface is expensive evidence |
| Corpus | 3,904,313 Stage-2 examples | 400k Stage-1 pool; 448k cached videos; Stage-2 mixture is not directly comparable | Do not equate cache size with SFT example count |
| Compute | 64 H800 GPUs, approximately 12 days | 8-GPU jobs | Reference result has much more optimization budget |
--max_frames; the trainer default remains 8. The code removes zero embeddings for t≥8 without training E_P on those chain lengths.| System | Dense | Codec | Delta | Token read |
|---|---|---|---|---|
| AdaCodec / Qwen3-VL-8B | 62.2 | 65.3 | +3.1 | 15.4% measured overall; 62.7 at ~1/7 tokens |
| Molmo2 V4, MLVU n=200 | 75.0 | 52.0 | −23.0 | 33% in the evaluated setup |
| Molmo2 V4, 3000 steps | 75.0 | 50.5 | −24.5 | More Stage-2 steps did not improve the codec gap |
| Gate | Change | Budget | Pass criterion | Why this order |
|---|---|---|---|---|
| Gate A | Unit-test block mask: patch states must be invariant to latent-query values at initialization | CPU / 1 GPU | Patch-state max error ≈ numerical noise | Cheap proof that the pretrained patch stream is preserved |
| Gate B | 64 latent grid; export layers −3 and −9; route through Molmo attention_meanq + MLP | 1 GPU smoke | 16 P outputs in the same LLM feature distribution as I outputs | Repairs the dominant feature-contract mismatch |
| Gate C | Stage-1 target after the deployed native connector; include P connector parameters in Stage 1 | 5k Stage-1 pilot | held-out real < shuffled; plus connector-space cosine / norm parity | Makes proxy success relevant to Stage 2 |
| Gate D | Freeze all visual-side modules in Stage 2; train only LLM | 500 and 1500 steps | MLVU n=200 beats V4 52 by ≥5 points at matched tokens | Separates architecture gain from scale |
| Gate E | --max_frames 17; global γ; remove/recalibrate motion saturation | ablation after Gate D | deep-GOP quality flat across t; stable I/P distribution | Prevents secondary codec-front-end effects from masking the interface result |
| Scale | Only then run 130k Stage 1 + long Stage 2 | full run | matched-budget MLVU, MVBench, VideoMME; quality vs GOP depth; TTFT/memory | Spend full compute only after the causal gates pass |