2 · WildTrack3D: current model architecture

Video + optional depth → tracking with memory → per-object 2D and 3D boxes

implementedquality rerun pendingoptional depth

Current project overview

Model architecture

(a) Native object tokens feed both 2D and 3D prediction. (b) Final-stage box geometry writes spatial memory; optional raw patch depth is retained separately for later attention. Open full-size vector figure.

First-frame boxes initialize the tracked objects; camera calibration supports camera-frame 3D predictions.
The diagram shows a propagation step. Only past frame records are read; the new record becomes available to the next frame.
Memory thumbnails summarize spatial features, object pointers and depth; colors are illustrative. The four-video overfit report evaluates an earlier revision.
Download vector PDF · Download SVG

Implementation details — expand as needed

Full data flow and implementation status

Current architecture is implemented and tested. The latest overfit videos were generated by the earlier architecture; they do not validate convergence of this revision.

Follow RGB, optional depth and historical memory to the object tokens, then to 2D and 3D outputs. Only the final stage writes the new memory record. Open full-size vector figure.

What is shared

Component	Current contract
3D head	Same Det3DHead class and predict_box3d_stages function; separate image/video weights
Video queries	Native selected decoder object tokens before pointer projection; removed the earlier extra query embedding/patch-attention path
Auxiliary supervision	Video: 2 native stages / 1 auxiliary. New joint image config: 6 stages / 5 auxiliary; legacy image default preserved
Memory	SAM causal multiplex tracking; bucket padding is never a target; only final stage writes memory
Detection-specific logic	Matching, O2M and text classification remain on the image path
Scope	First-frame prompted objects; no automatic object births or global re-identification

Depth and spatial memory

Each historical record contains image features, bucket memory and pointers. Optional raw depth remains one shared map; colors are schematic. Open full-size vector figure.

Select a historical frame and object; inspect occupancy and prompting metadata separately.
Open the interactive figure at full size

The optional-depth path constructs no LingBot backend and predicts no dense depth map. Missing depth is an explicit input condition.
Persistent state stores image features, object-conditioned spatial memory, object pointers, frame positions and a separate shared depth map.
Depth is not copied per object and is not warped onto the next frame. Attention learns correspondence; this is not an explicit world map.
One raw 72 × 72 FP32 map costs 20,736 bytes per retained frame; this excludes learned memory and attention activations.
Depth projections are bias-free and zero initialized; the original visual projection weights and attention width remain intact.
Monitoring logs projection weights, gradients and depth-to-base contribution RMS. There is no learned scalar gate; nonzero contribution alone does not prove improved accuracy.

Training, labels and losses

Image and video keep separate decoder/head weights while using common 3D prediction and loss functions. Open full-size vector figure.

Video input: RGB sequence, first-frame normalized xyxy boxes, per-frame intrinsics, optional metric patch depth and stable object IDs. Outputs: 2D boxes, presence, camera-frame 3D boxes and quality.
3D dimensions are WLH; local x/y/z extents are L/H/W. Quaternions are wxyz. JSON already uses WLH; trajectory NPZ LHW requires explicit reordering.
Raw metric scale is applied exactly once in the cache. Missing cameras disable the affected 3D supervision; unknown visibility is not absence.
2D losses use cxcywh L1 and xyxy GIoU. 3D regression uses center-offset, depth, dimensions and rotation, with the original WildDet quality helper.
Video target encoding uses the approved 50/50 GT versus detached predicted 2D anchor policy. This is not memory teacher forcing.
Training depth dropout is one dataset decision: 50% keep, 25% whole-clip missing, 25% contiguous gap. Diagnostic overfit disables it.
Known limitation: the inherited 3D IoU quality helper retains its legacy geometry approximation. Per-frame canonical rotation can also introduce temporal representation jumps.

Verification and version boundary

Evidence	Result / interpretation
Project suite	315 passed
Focused alignment suite	53 passed; overlaps the project suite
H200 feature-level suite	7 passed; includes 1008-grid, 17 objects and gradients through historical memory
Archive format	wildtrack3d_native_query_depth_v4; rejects old custom-head archives
Code revision	827b3a2; contract in docs/SHARED_3D_HEAD_PROPOSAL_20260917.md
Training quality	No new overfit or held-out quality result for this revision

Implementation tests validate interfaces, parity, gradients, serialization and task routing. They are not an accuracy or throughput benchmark.

Updated 2026-09-17 · Consolidated from saved experiment artifacts and inspected implementation · Historical URLs remain available with archive notices.