Reka’s RIDM Labels Real Video With Game-Trained Camera Motions

On October 2, 2026, Reka open-sourced the Reka Inverse Dynamics Model (RIDM)—Apache-2.0 weights and inference code that recover low-level camera and movement commands from short video clips, transferring from game captures to real-world egocentric footage. The company blog and Hugging Face card frame RIDM as plumbing for interactive world models: once unlabeled walks and turns carry predicted W/A/S/D/Shift keys plus yaw and pitch, those clips can supervise world-language-action systems that must respond to low-level controls rather than only to text.

From game labels to unlabeled walks

Interactive world models sit between classic robotics simulators and generative video. Simulators often look too clean; pure video generators may look real without obeying motor commands. Reka’s pipeline uses RIDM as an automatic captioner for motion: train an inverse-dynamics model on engine-labeled gameplay, run it on real video that has no action labels, then train a world model on the resulting video-plus-action pairs. The shipped artifacts are the IDM stage—two compact networks, not a full world model.

Architecture choices emphasize transfer. The flow model freezes Torchvision’s RAFT-small optical-flow front end (~0.99M parameters), then trains a small transformer stem (~1.8M trainable; ~2.8M total) over short five-frame windows so later layers never see raw pixels—only motion fields. The pixel model is fully trainable (~9.8M) with an IMPALA-style trunk and a 150-frame residual-attention window, closer to VPT-style policy backbones. Both heads emit five key probabilities (W, A, S, D, Shift) plus continuous yaw and pitch. Reka argues optical flow looks more alike between corridors and streets than RGB textures do, which is why the flow path is the recommended default for real walks.

Flow versus pixel inverse dynamics

Training data is engine-annotated gameplay, not hand labels: thousands of recorded plays at 1280×720 and 48 fps with eleven ground-truth key commands, mouse movement, and exact camera angles per frame. Resampling balances rare strafe, back, and tilt motions against dominant forward motion. Evaluation mixes real sensor-labeled video, GoPro clips, and held-out game footage. On Reka’s tables, the flow model reaches about 85.9% on real-video forward/turn classification (78 clips) versus about 30.8% for the pixel model, and about 91.5% versus 51.1% on real turn-direction subsets. Gaming footage gaps are smaller but still favor flow on most rows. Failures called out include false turns on static or dark frames, confusion between side-step and opposite yaw, walk keys on zoom-only cameras, and action-cam stabilization hiding rotation.

The HF repository keeps the two variants separable: flow/model.safetensors (~7.2 MB) and pixel/model.safetensors (~39.4 MB), each with config.json and inference.py, plus smoke tests and sample clips. Install is hf download then a one-liner over a clip; walking footage uses a different config and larger stride than game defaults. License text is careful: Reka weights and code are Apache 2.0, but RAFT-small is BSD-3 from Torchvision and is not bundled; pretrained RAFT training-data terms may restrict some commercial flow-model uses, and demo videos in assets/ carry CC BY 3.0 overlay terms. Pixel RIDM needs no third-party weights.

Weights, RAFT caveats, and smoke tests

For robotics and world-model teams, RIDM is a narrow but useful open brick: cheap motion labels from egocentric video without instrumenting every camera with IMU ground truth. It is not a navigation policy, not a physics engine, and not a claim that game-trained IDs replace domain adaptation. Operators should validate on their camera stack—stabilized GoPros and zoom-heavy phones are known weak spots—and keep humans in the loop before trusting labels for safety-critical robot training.

Practically, teams should start with the flow checkpoint and the walking config when labeling outdoor egocentric video, then spot-check windows where the model reports high Shift without translation or large yaw on nearly static frames. The pixel path remains useful as a research baseline for game-domain footage and for ablations that ask how much transfer optical flow actually buys. Because each folder runs alone, CI can smoke-test with pytest tests/test_smoke.py against examples/expected_flow_output.json. A full flow inference job still depends on Torchvision’s RAFT-small.

Compared with heavier vision-language action models, RIDM’s value is specialization and size: a few million parameters for motion captions rather than another general VLM call per clip. That economics matters when labeling thousands of hours before world-model training. Reka’s own failure gallery is unusually candid for a launch post—smoke screens, opposite yaw-versus-strafe cases, and zoom artifacts—so treat those clips as a minimal eval suite when porting to a new camera.

For Writeble’s opensource desk, RIDM is the October 2 robotics-adjacent release that couples a clear Apache primary with a concrete transfer story: learn controls in games, annotate the real world, then feed interactive world models.