NVIDIA Releases DeepStream 9.1 with Unified GitHub Monorepo and Agentic Video Analytics

NVIDIA has released DeepStream 9.1, transitioning its streaming video analytics software development kit (SDK) to a unified GitHub repository. The update introduces 11 new agentic skills, expanding the total number of agentic skills from 2 (in version 9.0) to 13, which enables developers to build and modify multi-camera video pipelines using natural-language prompts. This release also features Multi-View 3D Tracking (MV3DT) to track objects continuously across cameras and AutoMagicCalib to automate camera calibration directly from video feeds.

Consolidating DeepStream Under a Single Monorepo

Prior to this release, DeepStream development resources and packages were distributed across multiple isolated channels. DeepStream 9.1 consolidates these assets into a single monorepo on GitHub. While the open-source components in the repository are licensed under CC-BY-4.0 and Apache-2.0, the repository also distributes proprietary binary libraries—such as deepstream-binaries—and sample data, which are governed by the proprietary Software License Agreement for NVIDIA Software Development Kits.

Additionally, SDK packages in .deb and .tar formats are now distributed directly as GitHub Release assets, replacing the NVIDIA GPU Cloud (NGC) as the primary package distribution platform. Pre-built multi-architecture containers are still hosted on the unified NGC repository, but older auxiliary repositories have been archived and will no longer receive updates. These legacy repositories include:

• deepstream_reference_apps
• deepstream_tao_apps
• deepstream_tools
• deepstream_dockers

All subsequent updates will be published exclusively in the unified monorepo. As part of this consolidation, the graphical user interface tool Graph Composer, which was deprecated in DeepStream 8.0, is completely removed and no longer offered starting with DeepStream 9.1.

Building Pipelines with Natural-Language Coding Agents

The modular skill architecture in DeepStream 9.1 is designed to interface with external artificial intelligence coding agents. Developers can construct and deploy multi-camera video processing loops by issuing plain-English commands to coding assistants such as Claude Code, Codex, or Cursor.

Instead of manually writing custom configuration files or stitching together lower-level code, developers describe their pipeline requirements in natural language. The coding agent interprets these prompts to automatically configure, coordinate, and launch the target video analytics setup.

To facilitate this, the SDK provides 13 specific agentic skills that act as modular APIs for AI coding agents. NVIDIA states that automating these configurations with natural-language instructions can reduce pipeline development timelines from weeks to hours.

Multi-View 3D Tracking and AutoMagicCalib

Deploying multi-camera systems often presents tracking challenges, as standard 2D tracking methods typically lose continuity when an object moves out of a single camera’s field of view. The newly included MV3DT skill resolves this by projecting 2D camera detections from multiple viewpoints into a shared 3D coordinate system.

The tracking system matches target paths in real-time by sharing tracked segments over MQTT and calculating their physical proximity in 3D space. This allows the system to assign and maintain a single globally consistent ID for a specific object. The mathematical and association approaches used in MV3DT are based on the methodology detailed in the paper “Fully Distributed Multi-View 3D Tracking in Real-Time.” Out of the box, the MV3DT pipeline supports three detector models:

• PeopleNetTransformer
• PeopleNet v2.6.3
• RT-DETR 2D

The RT-DETR 2D model is pre-configured to detect pedestrians, transporters, and forklifts. Processed tracker outputs are delivered as an on-screen visual overlay, a birds-eye view trajectory map, and structured message data sent via Kafka protobuf metadata.

To project 2D detections into 3D coordinates, the system requires precise camera calibration parameters. Traditionally, this setup requires manual intervention to measure lens distortion, physical camera height, and angles using checkerboard calibration grids. The AutoMagicCalib (AMC) skill automates this step by analyzing synchronized video streams from active cameras to estimate both intrinsic parameters (such as focal length) and extrinsic spatial coordinates. Operating on a ground-plane assumption, AMC generates the 3x4 projection matrix necessary for 3D mapping directly from standard video feeds.

Core GStreamer Pipeline Architecture

DeepStream 9.1 continues to leverage the open-source GStreamer multimedia framework, structuring video processing as a sequence of connected, hardware-accelerated plugins. In a typical video analytics pipeline, a video stream is received from a source—such as a USB camera, a CSI camera, or an RTSP stream—and moves sequentially through several processing blocks:

1. Decode: Decompressing the incoming raw video stream using hardware decoders.
2. Pre-process: Scaling, color space conversion, and normalizing the frames.
3. Primary Inference: Running object detection models to identify targets like vehicles or people.
4. Tracking: Associating identified objects across sequential frames to create continuous tracks.
5. Secondary Inference: Running classification models on detected objects, such as identifying a vehicle’s color or make.
6. On-Screen Display (OSD): Drawing bounding boxes and text overlays on the video.
7. Sink: Outputting the processed stream to a display, a file, or streaming it over a network.

DeepStream maintains high throughput by retaining video frames directly in GPU memory as they pass between plugins. This zero-memory-copy architecture avoids the latency overhead associated with transferring heavy video data between CPU host memory and GPU memory.

Hardware Compatibility and Deployment Constraints

DeepStream 9.1 extends platform compatibility to newer edge architectures through integration with NVIDIA JetPack 7.2. This integration supports deployment across the NVIDIA Jetson edge AI product line, including both the Orin and Thor architectures.

For data center and enterprise applications, the SDK supports x86 and ARM SBSA platforms. Compatible enterprise GPUs include:

• NVIDIA T4
• NVIDIA Hopper
• NVIDIA Ampere
• NVIDIA Ada
• NVIDIA Blackwell

To simplify initial host configuration, this version introduces Ansible playbooks to provision Ubuntu hosts for x86 and ARM SBSA architectures. The SDK officially supports Ubuntu 24.04 LTS, though enterprise deployments using NVIDIA AI Enterprise software are limited to x86 platforms.

Certain technical limits remain in place on Jetson edge platforms. Applications utilizing V4L2 codecs are capped at a maximum of 1,024 combined decode and encode instances, although developers can modify this ceiling within the open-source code. Additionally, for deployments using gst-dsexample on Jetson, the detected-min-w and detected-min-h values must be set to values larger than 32 pixels in the primary inference configuration file (config_infer_primary.txt), rather than in the configuration files for gst-dsexample itself.