← Back to project

1 · Box tracker: design, training and final evaluation

Stage 1 · mechanism, training recipe and selected full-validation result

2D trackingbox memoryselected checkpoint

Current project overview

Selected result

Stage 1 delivers a 2D multi-object box tracker with recurrent box memory. The selected inference policy preserves the last valid reference when an object is absent.

Full SA-V validation · 1008 px	Visible IoU	Success@0.5	Presence F1
Mask → box	55.58%	59.71%	83.87%
Box raw	54.10%	59.94%	86.45%
Box + hold reference	55.46%	61.97%	86.89%

The selected policy is 0.12 percentage points below mask-to-box IoU, with higher success and presence F1.

Stage 1 demo · box memory versus mask memory

Use Geometry → Box student / Mask teacher to compare rectangle and silhouette inputs for the same scene.
Move the frame slider and select an object to inspect the historical memory record and its object slots.
This is a structural illustration. Feature colors are schematic; actual tracking predictions are shown in the videos below.

Select a historical frame and object; inspect occupancy and prompting metadata separately.
Open the interactive figure at full size

Stage 1 tracking videos · all 32 original comparisons

All 32 original comparison videos are restored here. These are early-checkpoint demonstrations, not videos of the final phase-30 + hold-reference checkpoint.

Left: box tracker, 1008 px. Right: mask→box baseline, 672 px. This archived comparison uses different resolutions.
Dashed boxes: ground truth. Solid boxes: predictions. Colors identify objects; p is per-object presence probability.
Playback is at real-time speed; tracking runs on annotated 6 FPS frames. Controls and fullscreen are available on every clip.
Archived student: v3_1008_geomkd/student_1008.pt. The final 155-video result above is a separate evaluation.
Original report with per-frame IoU curves and provenance · Per-video metrics

sav_000262

Visible IoU: box 70.4% · mask→box 32.4%
Open video at full size

sav_001676

Visible IoU: box 43.4% · mask→box 28.9%
Open video at full size

sav_001799

Visible IoU: box 60.0% · mask→box 71.8%
Open video at full size

sav_001826

Visible IoU: box 35.9% · mask→box 38.1%
Open video at full size

sav_001877

Visible IoU: box 15.3% · mask→box 22.3%
Open video at full size

sav_001964

Visible IoU: box 70.2% · mask→box 79.2%
Open video at full size

sav_002534

Visible IoU: box 41.7% · mask→box 50.2%
Open video at full size

sav_003501

Visible IoU: box 94.6% · mask→box 99.0%
Open video at full size

sav_004164

Visible IoU: box 60.4% · mask→box 84.3%
Open video at full size

sav_004380

Visible IoU: box 24.1% · mask→box 25.6%
Open video at full size

sav_004844

Visible IoU: box 27.8% · mask→box 52.6%
Open video at full size

sav_004873

Visible IoU: box 39.8% · mask→box 39.9%
Open video at full size

sav_005270

Visible IoU: box 66.6% · mask→box 59.2%
Open video at full size

sav_005574

Visible IoU: box 57.1% · mask→box 95.3%
Open video at full size

sav_005961

Visible IoU: box 74.8% · mask→box 89.4%
Open video at full size

sav_006114

Visible IoU: box 86.4% · mask→box 89.9%
Open video at full size

sav_006251

Visible IoU: box 95.8% · mask→box 98.8%
Open video at full size

sav_006298

Visible IoU: box 77.6% · mask→box 85.3%
Open video at full size

sav_006493

Visible IoU: box 45.5% · mask→box 40.4%
Open video at full size

sav_006751

Visible IoU: box 76.2% · mask→box 79.4%
Open video at full size

sav_006941

Visible IoU: box 42.8% · mask→box 47.7%
Open video at full size

sav_009079

Visible IoU: box 4.1% · mask→box 51.6%
Open video at full size

sav_009503

Visible IoU: box 55.5% · mask→box 77.1%
Open video at full size

sav_009628

Visible IoU: box 29.5% · mask→box 36.9%
Open video at full size

sav_010291

Visible IoU: box 71.1% · mask→box 35.3%
Open video at full size

sav_010406

Visible IoU: box 31.1% · mask→box 27.5%
Open video at full size

sav_010810

Visible IoU: box 54.1% · mask→box 61.5%
Open video at full size

sav_010922

Visible IoU: box 59.7% · mask→box 45.0%
Open video at full size

sav_011301

Visible IoU: box 45.5% · mask→box 57.0%
Open video at full size

sav_011413

Visible IoU: box 43.5% · mask→box 75.7%
Open video at full size

sav_011732

Visible IoU: box 53.2% · mask→box 57.8%
Open video at full size

sav_011944

Visible IoU: box 84.4% · mask→box 68.3%
Open video at full size

Box memory versus mask memory

Silhouette and rectangle occupancy provide different spatial shapes for the same objects. Open full-size vector figure.

Prompt metadata is a constant map per object. A learned encoder maps 32 input channels to 256 feature channels before addition to image features. Open full-size vector figure.

Component	Mask teacher	Selected box student
Object output	Segmentation mask	Detector-reference 2D box and presence
Memory geometry	Mask probability	Differentiable rectangle occupancy, gated by presence
Encoder weights	Official SAM modules	Initialized from the teacher; trained downsampler, image projection, Fuser and output projection
Fusion	Projected image features + encoded geometry	Same additive fusion, followed by Fuser
Background	Spatial feature map with object conditioning	Full spatial feature grid remains; outside-box features are not forcibly zeroed
Multi-object state	Multiplex object slots and pointers	Same mechanism; 16 slots per bucket, additional buckets for more objects

The rectangle is a geometric rasterization of four coordinates; it is not a learned segmentation output.
Foreground channels are packed first, then conditioning channels, matching the official input layout.
At 1008 input, the retained feature grid is 72 × 72. The geometry downsampler reaches this grid; the Fuser does not further reduce it.

Training recipe

Setting	Selected phase-30 run
Data	SA-V manifest: 50,583 records; sampling/retries/padding do not imply one unique successful update per record
Batch and clip	8 GPUs; one 8-frame clip per GPU; up to 16 sampled initial objects per clip; 1008 × 1008
Schedule	3 data epochs split into 10 phases each; checkpoint phase 30. This is not 30 complete data epochs.
Learning rate	2 × 10⁻⁶ base LR
Objectives	GT box 5; teacher-box KD 5; GIoU 2; GT presence 1; presence KD 1; memory 1; pointer 0.1
Frozen / trainable	Frozen RGB backbone and teacher; trainable student decoder, rectangle encoder, memory attention and pointers

The training sampler uses the configured random frame selection; the 6 FPS evaluation protocol below is not a claim that every training clip was fixed at 6 FPS.
Recurrent writes use predicted boxes/presence after initialization; the selected export clips boxes and retains the reference on absence.

The student learns both box outputs and recurrent memory behavior from the mask teacher, with matched object slots.

Final evaluation and provenance

Full validation: 155 videos; annotated 6 FPS frames; 23,132 visible and 8,542 absent object-frames.
First-visible objects are initialized once; initialization is excluded from scoring; no later corrections. This is box tracking, not mask J&F.
These results do not establish minute-scale, open-world object births or out-of-domain 3D tracking performance.
Selected export: sam3.1-training/outputs/v3_1008_geomkd_cliptrain30/student_1008_clip30_holdref.pt.
Source commit: 054be1c; evaluation: outputs/eval_final_holdref_155/summary.json.

Secondary ablation on the same 155 videos	Visible IoU	Status
Initial-memory KD continuation	54.00%	Not selected
Long-clip 16-frame continuation	54.02%	Not selected

Updated 2026-09-17 · Consolidated from saved experiment artifacts and inspected implementation · Historical URLs remain available with archive notices.