Sushanth
← Back to Knowledge Base

Paper Review: Reconstructing Dynamic Scenes by Querying Points

A new shape for reconstruction networks. Encode the video once, then ask a lightweight decoder about one point at a time. D4RT in depth, Point4D in brief, and how you train one of these.

Sep 12, 2026

Introduction

Every feed-forward reconstruction model of the last two years has the same shape:

  • A big transformer looks at all the frames at once (DUSt3R, VGGT, $\pi^3$, MapAnything, Depth Anything 3).
  • A stack of DPT-style heads decodes every pixel of every frame into depth, a point map, a camera.
  • One head per output.

Dynamic scenes expose two problems with that shape:

  • Nothing in the interface says “at what time”. A point map is “where is this pixel, now”. There is no head for “where is this pixel three frames later”.
  • Everything is decoded whether you need it or not. A 48-frame clip at 256×256 is 3.1 M outputs per head, per forward pass. Even if all you wanted was one camera pose.
Three dynamic scenes (a swan on water, a flower, a model train) reconstructed by MegaSaM, pi-cubed, SpatialTrackerV2 and D4RT; MegaSaM repeats the swan, pi-cubed loses the flower, SpatialTrackerV2 leaves gaps, D4RT reconstructs all of it
Reconstruction models repeat the swan or lose the flower. A tracker leaves holes. Figure 4 from D4RT, arXiv:2512.08924.

D4RT (Google DeepMind, CVPR 2026 oral) proposes a different shape: encode the video once, then answer questions about it with a small decoder, one point at a time. The query says which point, which time, which camera.

Four parts:

  1. D4RT: the query, and what it buys. Four released models are run on two toy clips so the difference is measured, not argued.
  2. Point4D (CMU, 2026): same interface, one change to the query so that it survives long videos.
  3. Training: which datasets, how much data, what supervision.
  4. Which one to reach for.

Part 1: Encode once, query anything (D4RT)

Four real models on two clips

Video 1 of 2. Two toy clips, four models, one score sheet. Every picture and number is a model output.

The clips. A small rendered world: a wall, a pillar, three posts, a floor. 80 frames each, 384×216. Two balls roll to the right (radius 0.28): the yellow one far and slow, the blue one near and fast.

  • Clip 1, fixed camera. Only the balls move. The blue ball overtakes the yellow one behind the pillar: the yellow ball is hidden in frames 15–35, the blue one in 20–30. Both stay inside the picture.
  • Clip 2, moving camera. The same overtaking while the camera drives 3.7 units to the right. Hidden frames 15–32 and 20–29.
Clip 1, five of the 80 rendered frames: two balls, a pillar, the blue ball passing behind it
Clip 1: frames 0, 20, 30, 40 and 79.
Clip 2, the same five frames with the camera moving to the right
Clip 2, the same frames. The camera moves with the balls.

The models. Each gets the 80 pictures and nothing else.

  Depth Anything 3 (1.4 B) VGGT-1B MapAnything Point4D
Output per picture depth, camera depth, point map, camera metric depth, rays, camera a 3D point per query, at every frame
Point tracks none: + CoTracker3 its own track head none: + CoTracker3 the queries themselves
Resolution 392×224 518×294 518×294 518×294, two chunks of 48 sharing 8

Two of the four have no tracking head. To get a point track out of Depth Anything 3 or MapAnything you run a 2D tracker (CoTracker3 here) and lift its pixel with the model’s own depth, intrinsics and camera of that frame. VGGT’s track head is lifted the same way. Point4D is asked directly: each ball’s pixel in frame 0, plus 576 static pixels on a grid.

Clip 1, what they returned.

Clip 1: the 80 depth maps of Depth Anything 3, VGGT and MapAnything each placed in one world with the model's own cameras; the static scene fuses, each ball is a row of blobs with a gap at the pillar
Clip 1: each dense model's 80 depth maps in one world, with its own cameras. Dashed white: the true paths.
Clip 1, Point4D: one thick line per ball through all 80 frames over the dashed truth, thin time-coloured lines for the static queries
Clip 1, Point4D: one line per ball, over the dashed truth. Thin lines: the static queries, which should be dots.
Clip 1 from above, four panels: the lifted ball tracks of Depth Anything 3 with CoTracker3, VGGT with its track head, MapAnything with CoTracker3, and Point4D, each over the dashed true paths
Clip 1 from above: each pipeline's two ball tracks over the dashed truth.

Clip 2, the camera moving.

Clip 2: the same three fused clouds; VGGT's is smeared because its camera path is a quarter of the true one
Clip 2. VGGT's cloud smears: it moved its camera 0.27× the true distance.
Clip 2 from above: the dense tracks dip to the pillar while a ball is hidden; Point4D's lines drift after the seam at frame 40
Clip 2 from above. The lifted tracks dip to the pillar while a ball is hidden. Point4D's lines drift after the seam.
Clip 2, Point4D: the static queries streak instead of staying put, the two chunks disagree
Clip 2, Point4D. The static queries should be dots. They streak: the second chunk does not sit on the first.

The numbers. The same definitions for every model, toy units, ball radius 0.28. Depth: AbsRel on the static surfaces after one global scale. Camera: ATE after placing the whole output once (frame 0 at its true pose, rotation fitted on all frames, the depth scale). Tracks: how far the answer lies outside the ball, $\max(0,\ \lVert p - c\rVert - 0.28)$, so 0 means on or inside it. Each cell is fixed camera / moving camera.

fixed / moving camera Depth Anything 3 VGGT MapAnything Point4D
Depth of a frame, static AbsRel 1.3 % / 0.8 % 0.6 % / 1.5 % 1.5 % / 2.4 % 2.3 % / 3.6 %
Scale, true / predicted 1.09 / 1.22 6.85 / 7.32 0.88 / 0.95 25.42 / 37.08
Camera ATE 0.01 / 0.06 0.00 / 0.67 0.01 / 0.52 0.08 / 0.80
Camera motion, predicted / true (moving clip) 1.04× 0.27× 0.81× 1.09×
fixed / moving camera Depth Anything 3 VGGT MapAnything Point4D
Point tracks from CoTracker3 own track head CoTracker3 3D queries
Yellow ball, visible: outside the ball 1.13 / 0.01 2.40 / 1.52 1.03 / 0.36 0.04 / 0.58
Yellow ball, hidden: outside the ball 1.71 / 1.49 1.53 / 1.59 1.74 / 1.79 0.11 / 0.01
Blue ball, visible: outside the ball 0.04 / 0.01 2.78 / 2.26 0.01 / 0.38 0.03 / 0.35
Blue ball, hidden: outside the ball 0.15 / 0.38 1.75 / 1.90 0.21 / 0.77 0.09 / 0.22
2D error, visible, px 14.9 / 1.8 142.1 / 58.2 14.9 / 1.8 6.0 / 38.3
2D error, hidden, px 5.1 / 1.2 100.0 / 99.4 5.1 / 1.2 4.0 / 11.5
Static queries, drift by the last frame - - - 0.15 / 0.95
Yellow: far, hidden for 21 frames. Blue: near, hidden for 11, and only 0.4 behind the pillar's front face.
RTX 5080, 80 frames Depth Anything 3 VGGT MapAnything Point4D
Resolution 392×224 518×294 518×294 518×294
Forward pass 4.6 s 7.2 s 4.7 s 14.2 s
2D tracker 0.6 s in the pass 0.6 s -
Time to point tracks 5.2 s 7.2 s 5.3 s 14.2 s
Peak GPU memory 11.7 GB 14.1 GB 11.8 GB 11.5 GB
Clip 1, one forward pass each. Point4D's backbone is Depth Anything 3 itself, run once per chunk, in bf16 to fit the card.

What falls out.

  • Per-frame geometry is not the problem. All four are within 4 % on both clips.
  • The moving camera separates the dense models. Depth Anything 3 gets the path (1.04×, ATE 0.06). MapAnything gets 0.81× of it. VGGT moves its camera 0.27× the true distance, and its fused cloud smears. A textureless wall and a checkerboard floor are a hard case.
  • A hidden ball gets the occluder. Every dense pipeline lifts the pillar: 1.5–1.8 outside the yellow ball, which sits 1.8 behind it. It is not a wrong answer about the ball. It is the pixel’s answer.
  • The 2D tracker fails in the open too. On clip 1 CoTracker3 loses the yellow ball behind the pillar, never re-acquires it (it reports the ball hidden for the remaining 44 frames) and ends 46 px off. On clip 2 it tracks both balls to within 2 px. VGGT’s track head never leaves the query pixel on either clip.
  • Point4D answers through the occlusion. 0.11 and 0.09 outside the ball on clip 1, 0.01 and 0.22 on clip 2, at a confidence of about 70.
  • Point4D’s failure is the seam. On clip 2 the second chunk (frames 40–79) does not sit on the first: the yellow answer is 0.02 outside in frames 20–39 and 1.64 outside in frames 70–79, and the static grid drifts by 0.95. On the fixed camera the drift is 0.15. That is Part 2’s subject.
  • It is not faster here. 14 s against 5–7 s. Encode once, decode per query is cheap only when the encoder is cheap; Point4D’s encoder is a 1.4 B backbone run once per chunk, and the dense models run theirs once. D4RT’s 200 FPS claim (below) is for a pose from a few hundred queries on an A100.

The query

flowchart LR
  V["video V<br/>T frames"] --> E["encoder E<br/>video ViT, runs once per clip"]
  E --> F["global features F<br/>N tokens x C"]
  Q["query q = (u, v, t_src, t_tgt, t_cam)<br/>+ the 9x9 patch around (u, v)"] --> D["decoder D<br/>8 cross-attention layers<br/>one query at a time"]
  F --> D
  D --> P["P = (x, y, z)<br/>+ 2D point, visibility, confidence"]
The encoder runs once per clip. The decoder runs once per query.
\[F = \mathcal E(V), \qquad q = (u,\, v,\, t_\text{src},\, t_\text{tgt},\, t_\text{cam}), \qquad \mathbf P = \mathcal D(q, F) \in \mathbb R^3.\]

Five numbers in, three numbers out:

Slot Meaning
$(u, v)$ a pixel, in normalised $[0,1]^2$ coordinates
$t_\text{src}$ the frame that pixel lives in
$t_\text{tgt}$ the time at which you want to know where that point is
$t_\text{cam}$ the camera whose coordinate system the answer is expressed in

The three time indices need not agree. That is the whole trick: which slots you hold fixed turns one decoder into every 4D task.

Task $u$ $v$ $t_\text{src}$ $t_\text{tgt}$ $t_\text{cam}$
Point track fixed fixed fixed $1 \dots T$ $= t_\text{tgt}$
Point cloud (one shared frame) $1 \dots W$ $1 \dots H$ $1 \dots T$ $= t_\text{src}$ fixed
Depth map $1 \dots W$ $1 \dots H$ $1 \dots T$ $= t_\text{src}$ $= t_\text{src}$
Extrinsics coarse grid coarse grid fixed $= t_\text{src}$ $1 \dots T$
Intrinsics coarse grid coarse grid $1 \dots T$ $= t_\text{src}$ $= t_\text{src}$
Table 1 of the paper, rewritten. "Fixed" means one value; a range means you sweep it.
  • A track: fix $(u, v, t_\text{src})$, sweep $t_\text{tgt} = t_\text{cam}$.
  • A point cloud in one frame: every pixel of every frame, $t_\text{cam}$ pinned to one camera. The change of reference frame happens inside the model.
  • A depth map: $t_\text{src} = t_\text{tgt} = t_\text{cam}$, keep the $z$ of $\mathbf P$.

Cameras fall out of the point sets

flowchart LR
  G["coarse pixel grid<br/>in frame i"] --> Qi["q = (u, v, i, i, i)<br/>answers in camera i"]
  G --> Qj["q = (u, v, i, i, j)<br/>answers in camera j"]
  Qi --> U["Umeyama<br/>one 3x3 SVD"]
  Qj --> U
  U --> Pose["relative pose R, t<br/>camera i to camera j"]
  Qi --> Fo["per point: f = p_z (u - 0.5) / p_x<br/>median over the grid"]
  Fo --> K["focal length<br/>(intrinsics)"]
No camera head. Extrinsics and intrinsics are both read off answer sets.
  • Extrinsics. Query a grid of frame $i$ twice, in camera $i$ and in camera $j$. The two answer sets are the same points in two frames; the rigid transform between them is the relative pose. One $3 \times 3$ SVD.
  • Intrinsics. Query a grid with all three times equal, assume a pinhole with the principal point at $(0.5, 0.5)$; every point gives a focal length, take the median. A fisheye needs a refinement on top (see the wide-FOV post).

What’s inside a query token

flowchart TB
  UV["(u, v)<br/>Fourier features"] --> SUM["query token = sum of five"]
  TS["t_src<br/>learned embedding"] --> SUM
  TT["t_tgt<br/>learned embedding"] --> SUM
  TC["t_cam<br/>learned embedding"] --> SUM
  PA["9x9 RGB patch at (u, v) in frame t_src<br/>small MLP"] --> SUM
  SUM --> DEC["cross-attention into F<br/>queries never see each other"]
  DEC --> OUT["13 numbers: xyz, uv, visibility,<br/>motion, normal, confidence"]
Figure 7 of the paper as a diagram. Only the first three of the 13 outputs are the point.

The patch looks like a detail and is not:

ViT-L on Sintel AbsRel (scale) ↓ ATE ↓
without the local patch 0.366 0.173
with the 9×9 patch 0.302 0.091
Table 7 of the paper. The patch halves the pose error.
  • Without it the decoder has to locate a sub-patch pixel from a 16×16-patch token grid alone.
  • Because $(u, v)$ is continuous you can query at the original video resolution while the encoder saw 256×256.
  • Queries never see each other. The authors tried self-attention between queries: “major performance drops”. This is what makes decoding embarrassingly parallel.

It is very fast

Each query is one cheap cross-attention pass. Cost scales with how many points you ask about, not with $T \times H \times W$.

Pose accuracy versus throughput on an A100: D4RT at 200+ FPS and 0.9 accuracy, pi-cubed at ~15 FPS, VGGT ~25 FPS, STv2 ~2 FPS, MegaSaM ~1 FPS, MapAnything ~5 FPS
A camera pose needs only a coarse grid of queries: 200+ FPS, 9× VGGT, 100× MegaSaM. Figure 3 from D4RT.
Full-video 3D tracks sustained at 60 FPS 24 FPS 10 FPS 1 FPS
DELTA 0 5 408 5,770
SpatialTrackerV2 29 84 219 2,290
D4RT 550 1,570 3,890 40,180
Table 3 of the paper, single A100.

Tracking every pixel without $T^2 HW$ queries

Video 2 of 2. Algorithm 1: how many tracks each source frame really needs.

A track for every pixel of every frame is $O(T^2 HW)$ queries. Algorithm 1 is the fix:

  • Keep a $T \times H \times W$ occupancy grid. Start tracks only from pixels nobody has visited yet.
  • Every decoded track marks all the pixels it visibly passes through as done.
  • Frame 0 starts a track from every pixel; those already cover most of every later frame. After that only new content spawns tracks.
  • The paper measures 5–15× fewer tracks; the toy world above gets 24×.

What D4RT buys, and what it costs

  • One decoder for everything. Tracks, point clouds, depth, intrinsics and extrinsics are query patterns, not heads.
  • Sparse or dense at will, at cost proportional to what you ask for.
  • Dynamic correspondence, which none of the pure reconstruction models provide.
  • The encoder sees a whole clip at once. Training is 48 frames; a long video has to be chunked, and the paper does not chain tracks across chunks.
  • The query is a pixel. It has to be visible in the source frame. That breaks at a chunk boundary. Part 2.
  • No code or weights.

Part 2: A pixel is a bad handle for a long video (Point4D)

Any encoder has a window, so a 300-frame video becomes overlapping chunks (48 frames, 8 of overlap). For static geometry that is solved: reconstruct each chunk, align the overlaps with a Sim(3). That is VGGT-Long.

A 300-frame corridor video split into four overlapping chunks; below, Point4D's continuous long-range tracks of a runner compared with TraceAnything, Any4D, 4RC and V-DPM, whose tracks break at chunk boundaries
Four chunks, three seams. Every pixel-query method loses the runner at one of them. Figure 1 from Point4D, arXiv:2609.09145.

For 4D it is not enough. A track has an identity that has to survive the boundary:

  • The track from chunk $k$ ends at a 3D point. To continue it you must re-query that point in chunk $k{+}1$.
  • A D4RT-style decoder only accepts pixels. So you project the endpoint into an overlap frame and ask again.
  • That fails exactly when long videos are interesting. Occluded: the pixel belongs to whatever is in front, and the track attaches itself to the occluder. Left the frame: there is no pixel at all.
  • Even for a visible point, the projection runs through an estimated camera.
Two panels. Left: 2D query-based chaining, where the predicted trajectory of chunk 1 is projected into an image of chunk 2 to obtain a new pixel query. Right: 3D query-based chaining, where the predicted 3D endpoint is used directly as the next query
Left, the 2D handoff: predict, project to a pixel, re-query. Right, the 3D handoff: predict, re-query. Figure 2 from Point4D.
Point4D, video 1 of 2. Two chunks, one seam, and a pixel handoff that lands on the pillar.

The fix: query points, not pixels

Point4D, video 2 of 2. The handoff as a Sim(3): the endpoint crosses the seam as three numbers.

Point4D’s query is

\[q = (\mathbf p,\; t_\text{src},\; t_\text{tgt},\; t_\text{cam},\; S), \qquad \mathbf p = (x, y, z):\]

where is the point at $\mathbf p$ (in camera $t_\text{src}$’s coordinates), whose appearance is $S$, at time $t_\text{tgt}$, in camera $t_\text{cam}$? The slots have D4RT’s names. Where each token comes from is different, and the released code shows it:

flowchart TB
  subgraph ENC["encoder output (Depth Anything 3, ViT-g): per frame"]
    direction LR
    ST["scene tokens F"]
    CT["camera token c_t"]
    TT["time token tau_t"]
  end
  XYZ["(x, y, z) in camera t_src<br/>Fourier encoding"] --> SUM
  CT -- "src_mlp(c at t_src)" --> SUM
  TT -- "tgt_mlp(tau at t_tgt)" --> SUM
  CT -- "cam_mlp(c at t_cam)" --> SUM
  S["9x9 patch from frame t_patch<br/>patch MLP, cut once and cached"] --> SUM
  SUM["query = sum / sqrt(5)"] --> DEC["cross-attention decoder"]
  ST --> DEC
  DEC --> OUT["x, y, z, u, v, confidence, visibility"]
New against D4RT: a 3D point instead of a pixel, time slots built from the encoder's own tokens, a frame index for the patch. From build_query_embed in point4d/heads/d4rt_head.py.

Three things follow:

  • The point is a 3D coordinate. Occluded and out-of-frame points are legal queries.
  • The appearance patch is cut once, where the query was born, and cached. A point hidden in chunk 5 still carries the look it had in frame 0.
  • The handoff is a coordinate transform. The overlap frames’ predicted depth gives a Sim(3) between chunks (Umeyama on a 1024-point grid, closest 85 % of pairs). The endpoint goes through it. No pixel, no matching, no visibility requirement.

Easy to miss: Point4D keeps Depth Anything 3’s dense depth and camera heads. The query decoder only answers motion questions.

Does it work?

The ablation of the query itself, on 200-frame sequences chunked 48/8:

Query Dynamic Replica EPE ↓ Survival ↑
2D pixel (D4RT-style), reprojection handoff 0.712 0.422
3D point, patch always from the source frame 0.825 0.266
3D point, patch from any visible frame 0.155 0.812
Table 3 of the paper. Survival: how much of the video a track lasts before its first failure.
  • The middle row matters. 3D coordinates alone do not help: a model trained only on visible source patches has never seen a query for a hidden point, and the handoff hands it exactly that. Both changes are needed.
  • By what the point was doing at the handoff (Table 7): occluded points 0.57 (best pixel method) against 0.20; visible 0.29 against 0.14; out of frame 0.34 against 0.31.
  • Caveats from the same paper. Within a single chunk Point4D is not the best decoder (4RC, V-DPM win); the gain is at the seams. Dense queries cost more than a DPT head. Everything rides on the predicted depth and the Sim(3), whose errors compound across chunks, which is exactly the drift in the table above.

In one line. D4RT turned reconstruction from “decode everything” into “ask about a point”; Point4D noticed that a point should be a coordinate, not a pixel.

Part 3: What it takes to train one

It is not just the decoder

  D4RT Point4D
Encoder ViT-g video transformer, ~1 B params Depth Anything 3 ViT-g, plus a time token per frame
Encoder initialisation VideoMAE DA3, including its depth head and camera decoder
Decoder 8 cross-attention layers, 144 M params, from scratch 8 cross-attention layers, from scratch
Encoder trained? Yes, end-to-end Yes, at 0.1× the learning rate
Released weights none one file of 5.8 GB, the whole network in fp32
  • The initialisation is not optional: from random weights, D4RT’s depth AbsRel goes from 0.30 to 0.74 and pose ATE from 0.09 to 0.33 (Table 11).
  • The honest description: a pretrained dense model, fine-tuned end-to-end, with a new query decoder grown on top.

The datasets

Dataset Kind Size Ground truth Share of each epoch
PointOdyssey synthetic 159 videos, ~200 K frames 3D point trajectories, depth, cameras 19.8%
Dynamic Replica synthetic 524 videos, 145 K frames long-range trajectories, depth, cameras 19.8%
BEDLAM 2 synthetic humans 27 K sequences, ~8 M frames SMPL-X bodies, cameras, depth 19.8%
CoTracker’s Kubric set synthetic 6 K sequences × 24 frames point tracks with visibility 11.9%
Kubric MOVi-F synthetic ~5 K videos × 24 frames depth, flow, object poses, cameras 11.9%
Waymo DriveTrack real ~10 K videos 3D tracks from LiDAR and 3D boxes 5.8%
OmniWorld mixed 600 K+ sequences depth, cameras; no trajectories 2.0%
ScanNet, ScanNet++, BlendedMVS, CO3Dv2 static large depth, poses 2.0% each
WildRGB-D real, static ~20 K object videos metric poses, depth, masks 1.0%
Point4D's training mixture (Table 4). D4RT: a similar mixture plus internal datasets.
  • The dynamic data is almost all synthetic. 83 % of an epoch is dynamic data with trajectory labels; every source is rendered except Waymo. Real video with 3D trajectory ground truth barely exists.
  • There is no self-supervision. An L1 between the predicted point and a ground-truth point, plus small 2D, visibility (and for D4RT motion, normal) losses. What replaces it is reuse: static datasets declare every point a stationary trajectory.
  • The query interface is what makes the mixture usable. OmniWorld has no trajectory labels, so its queries are restricted to $t_\text{src} = t_\text{tgt}$. Loss terms are applied only where a label exists.

How the queries are sampled

  D4RT Point4D
Queries per training clip 2,048 per 48-frame clip 750 per frame
Hard-example bias 30 % near depth or motion edges 40 % on Sobel edges
Time slots uniform; $t_\text{tgt} = t_\text{cam}$ with probability 0.4 uniform; $t_\text{cam} = t_\text{tgt}$ for 40 %
Extra weighting confidence-weighted L1 dynamic points upweighted

A dense head on a 48-frame clip at 256×256 is scored on 3.1 M outputs; D4RT on 2,048 decoder calls. Supervision cost does not depend on resolution, which is also why the decoder can later be queried at full resolution.

The recipe

  D4RT Point4D
Clip length 48 frames 16–64 frames
Resolution 256×256, with an aspect-ratio token width sampled in [252, 518]
Optimiser AdamW, wd 0.03, clip 10; warm-up 2,500 steps to $10^{-4}$, cosine AdamW; warm-up 10 epochs to $10^{-4}$, cosine; 0.1× for DA3 parts
Length 500 K steps 150 epochs
Hardware 64 TPU chips, “just over 2 days” 8 H100 GPUs
Losses $c\,\mathcal L_{3D} - 0.2\log c$ + 0.1 (2D, vis, motion) + 0.5 normal point + conf + 2D + vis, unit weights

So is it light to train?

Model Hardware Time
VGGT 64 × A100 9 days
Depth Anything 3 (giant) 128 × H100 ~10 days
D4RT 64 TPU chips ~2 days
Point4D 8 × H100 150 epochs

Yes, by a lot, but not because “only the decoder is new”. The saving comes from a pretrained initialisation and sparse supervision. Point4D at eight GPUs is possible because Depth Anything 3 already paid for the 128 H100s.

How they are evaluated

  • D4RT, as a general reconstruction model: 3D tracking on TAPVid-3D; point clouds on Sintel and ScanNet; video depth on Sintel, ScanNet, KITTI, Bonn; pose on Sintel, ScanNet, RealEstate10K; throughput on one A100.
  • Point4D, on the one thing it claims: long-video tracking on 200-frame PointOdyssey and Dynamic Replica and 150-frame Panoptic Studio, chunked 48/8, with end-point error, APD and survival; horizons to 500 frames; single-chunk tracking, where it is honest about not winning.
  • The caveat: PointOdyssey and Dynamic Replica are also the two largest training sources. Most of the evidence is on held-out synthetic splits.

Side by side

  Dense heads D4RT: 2D query Point4D: 3D query
What the decoder is DPT head per output one cross-attention decoder same decoder
The question you can ask “depth / points of this frame” “where is pixel $(u,v,t)$ at time $t’$, in camera $t’’$” “where is point $\mathbf p$ at time $t’$, in camera $t’’$”
Dynamic correspondence no yes yes
Cost scales with $T \times H \times W$ number of queries number of queries
Query must be visible in the source frame? n/a yes no
Chaining across chunks align geometry with a Sim(3) project endpoint to a pixel, re-query transform endpoint with the Sim(3), re-query
Where geometry comes from the heads the queries DA3’s dense heads
Training compute VGGT 64 A100 × 9 d; DA3 128 H100 × 10 d 64 TPU chips × 2 d 8 H100
Code VGGT, DA3, MapAnything not released released, weights on Hugging Face

Part 4: Which one should you reach for?

flowchart TD
    Q1["<b>Does anything in the scene move?</b>"]
    Q2["<b>Does the video fit one encoder window?</b><br/><i>~48–64 frames</i>"]
    Q3["<b>How many points do you need?</b>"]
    DH["<b>dense heads</b><br/>VGGT · DA3 · MapAnything<br/><i>depth, points, pose per frame</i>"]
    QD["<b>query decoder</b><br/>D4RT-style 2D queries<br/><i>cost ∝ points asked for</i>"]
    DD["<b>dense 4D heads</b><br/>4RC · V-DPM<br/><i>or D4RT's occupancy-grid tracking</i>"]
    P4["<b>3D query + chaining</b><br/>Point4D<br/><i>handoff is a Sim(3), not a pixel</i>"]

    Q1 -- "no" --> DH
    Q1 -- "yes" --> Q2
    Q2 -- "no, it is long" --> P4
    Q2 -- "yes" --> Q3
    Q3 -- "a few thousand tracks,<br/>or a pose" --> QD
    Q3 -- "every pixel, every frame" --> DD

    classDef q fill:#1a1d27,stroke:#6c9eff,color:#e0e0e6;
    classDef rec fill:#17301f,stroke:#2e8b57,color:#e0e0e6;
    classDef p4 fill:#2a2340,stroke:#8b5cf6,color:#e0e0e6;
    class Q1,Q2,Q3 q;
    class DH,QD,DD rec;
    class P4 p4;
Static scenes never needed a query. Long dynamic videos need the query to be a 3D point.

For a few hundred frames of handheld video with people walking in and out of view:

  1. You have to chunk, so the question is only how tracks cross the seams.
  2. The seams are where the occlusions are. Points that never disappear would not have needed a long-video method.
  3. Point4D is the one you can run. Same DA3 backbone as the static tools, weights on the hub, and the chaining loop is 250 lines of numpy around the model.

References

  • Zhang et al., Efficiently Reconstructing Dynamic Scenes One D4RT at a Time, CVPR 2026 — arXiv:2512.08924 · project page
  • Jeon, Karhade, Ramanan, Tulsiani, Point4D: Long-range 4D Motion Reconstruction, 2026 — arXiv:2609.09145 · code · weights · project page
  • Sajjadi et al., Scene Representation Transformer, CVPR 2022 — arXiv:2111.13152 (the decoder D4RT is modelled on)
  • Lin et al., Depth Anything 3: Recovering the Visual Space from Any Views, 2025 — arXiv:2511.10647 · code (Point4D’s backbone; DA3NESTED-GIANT-LARGE is the model run above)
  • Tong et al., VideoMAE, NeurIPS 2022 — arXiv:2203.12602 (D4RT’s encoder initialisation)
  • Wang et al., VGGT: Visual Geometry Grounded Transformer, CVPR 2025 — arXiv:2503.11651 · code
  • Deng et al., VGGT-Long: Chunk it, Loop it, Align it, 2025 — arXiv:2507.16443
  • Keetha et al., MapAnything: Universal Feed-Forward Metric 3D Reconstruction, 2025 — arXiv:2509.13414 · code
  • Karaev et al., CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos, 2024 — arXiv:2410.11831 · code (the 2D tracker paired with the dense models above)
  • Luo et al., 4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere, 2026 — arXiv:2602.10094
  • Sucar et al., V-DPM: 4D Video Reconstruction with Dynamic Point Maps, 2026 — arXiv:2601.09499
  • Xiao et al., SpatialTrackerV2, ICCV 2025 — arXiv:2507.12462
  • Zheng et al., PointOdyssey, ICCV 2023 — pointodyssey.com · Karaev et al., DynamicStereo (Dynamic Replica), CVPR 2023 — github · Balasingam et al., DriveTrack, CVPR 2024 — arXiv:2312.09523 · Zhou et al., OmniWorld, 2025 — arXiv:2509.12201
  • Umeyama, Least-squares estimation of transformation parameters between two point patterns, TPAMI 1991

All paper figures above are reproduced from the linked arXiv preprints and are credited in their captions. Code details are from the Point4D repository at the commit current on 17 September 2026.