NVIDIA Introduces Python-Based DLA Inference on Jetson AGX Orin with cuDLA

NVIDIA has introduced a Python-based workflow for running Deep Learning Accelerator (DLA) inference on its Jetson AGX Orin platform, leveraging the cuDLA API. This development aims to enhance power efficiency and free up GPU capacity for additional tasks in edge AI applications, where compute and power resources are often constrained.

cuDLA and Python Integration

The NVIDIA cuDLA runtime API manages execution on the DLA hardware, allowing applications to offload supported neural network operations from the GPU. The newly introduced cuda.bindings.cudla module, part of the CUDA Python project, provides a direct Python interface to the cuDLA C++ API. This integration enables developers to manage the DLA runtime and orchestrate inference workflows entirely in Python, eliminating the need for C++ code.

Introduced in CUDA bindings version 13.3.0, the cuda.bindings.cudla module supports hybrid-mode workflows. In this mode, CUDA streams coordinate DLA inference with GPU-based preprocessing and postprocessing tasks, ensuring efficient execution pipelines. The API exposes 13 functions, 9 data types, and 8 enumerations, facilitating seamless integration with existing CUDA workflows.

Inference Workflow on Jetson AGX Orin

The inference lifecycle on Jetson AGX Orin involves several steps:

• Initializing CUDA and creating a DLA device in hybrid mode.
• Loading a compiled DLA loadable file, which encodes a trained neural network in a format executable by the DLA hardware.
• Allocating GPU memory buffers, registering them with cuDLA, and submitting tasks through a CUDA stream.
• Synchronizing execution and retrieving inference results.

To compile a DLA loadable, developers use TensorRT's trtexec tool. Key flags such as --useDLACore=0 and --buildDLAStandalone target the first DLA engine and produce a standalone loadable, respectively. The resulting binary can be executed directly on the DLA hardware without requiring TensorRT at runtime.

Platform Compatibility and Installation

The cuDLA Python bindings are compatible with NVIDIA Jetson AGX Orin running JetPack 7.2.2 or later. The required runtime library, libcudla.so, is included with the JetPack installation. Developers can install the bindings using the following command:

pip install "cuda-bindings>=13.3.0"

Python 3.10 or later is required to use the bindings. If the default Python version on Jetson Orin is older, developers can install Python 3.12 from the deadsnakes PPA.

Expanding Beyond Embedded Platforms

While this release focuses on Jetson AGX Orin, the cuDLA API is also compatible with NVIDIA RTX Spark, bringing DLA inference capabilities to Windows-based AI developer laptops. The API remains consistent across platforms, enabling developers to apply the same workflows in both embedded and desktop environments.

This advancement underscores NVIDIA's commitment to providing flexible and efficient tools for AI inference, catering to a wide range of applications from robotics to edge computing and beyond.