The Cube of AI Power: A Hands-on Review of the NVIDIA Jetson AGX Orin Developer's Kit

Designed as the successor to the company’s Jetson AGX Xavier family of systems-on-module (SOMs), the NVIDIA AGX Orin is here — in developer kit form, at least — with the promise of major performance gains over its predecessor.

Where the AGX Xavier range was launched with the promise of “cloud native” capabilities, the AGX Orin has a broader goal in mind: ushering in what its creators call the “new age of robotics.” It comes claiming a big boost in performance for artificial intelligence and deep learning workloads — and serves as the flagship platform for JetPack 5.0, NVIDIA’s latest artificial intelligence and deep learning software development kit.

But does the compact cube deliver?

Hardware

• CPU: 12-core 2.2GHz NVIDIA Cortex-A78AE ARMv8.2 with 3MB L2 and 6MB L3 cache
• GPU: 1.3GHz NVIDIA Ampere with 2,048 CUDA cores and 64 Tensor cores
• Accelerators: 2× 1.6GHz NVIDIA Deep Learning Accelerator v2 (NVDLA v2), 1× Programmable Vision Accelerator v2.0 (PVA v2.0)
• RAM: 32GB LPDDR5 at 3,200MHz
• Storage: 64GB eMMC 5.1, microSD, M.2 Key M NVMe
• USB: 2× USB 3.2 Gen 2 Type-A, 2× USB 3.2 Gen 1 Type-A, 2× USB 3.2 Gen 2 Type-C with USB PD (one used for power input); 1× micro-USB 2.0 (debug)
• PCIe: Mechanical 16× with 8× PCIe Gen 4 lanes, 1× M.2 Key M with 4× PCIe Gen 4 lanes, 1× M.2 Key E with 1× PCIe Gen 4 lane (populated by wireless module)
• Networking: 10-gig-Ethernet, M.2 Key E 802.11ac 2x2 2.4/5GHz, Bluetooth 5.0
• Display Outputs: 1× multi-mode DisplayPort 1.4a, 8k60
• Camera Inputs: 1× 120-pin expansion connector: 16-lane MIPI CSI-2 with 16 virtual channels, D-PHY 2.1 (up to 40Gbps), C-PHY 2.0 (up to 164Gbps)
• GPIO: 40-pin header (populated) with UART, SPI, I2C, I2S, PWM, 12-pin automation header, 10-pin audio header, 10-pin JTAG header
• Video Encode (H.265): 2× 4k60, 4× 4k30, 8× 1080p60, 16× 1080p30; H.264, AV1 also supported
• Video Decode (H.265): 1× 8k30, 3× 4k60, 7× 4k30, 11× 1080p60, 22× 1080p30; H.265, VP9, AV1 also supported
• Dimensions: 110×110×71.65mm (around 4.3×4.3×2.82")
• Kit Contents: 1× Jetson AGX Orin Developer’s Kit in case with heatsink and fan, 1× AzureWave AW-CB375NF 802.11a/b/g/n/ac Wi-Fi and Bluetooth 5.0 M.2 2230 module (installed), 1× 90W USB PD Type-C power supply, 1x USB Type-A to Type-C cable, Quick Start Guide

On paper, the Jetson AGX Orin is an absolute beast — yet sits in a tiny 4.3×4.3" footprint, if you exclude the surprisingly hefty 90W USB PD external power supply bundled in the box. As with its predecessor, the AGX Xavier, there’s plenty of room for expansion including a full-size PCI Express slot — though depending on what you plug into it, you’ll likely need to swap out the bundled power supply for something with a little more headroom.

In its Developer’s Kit form, as reviewed, the Jetson AGX Orin comes pre-loaded with the latest Jetpack 5.0 software development kit — based, now, on Canonical’s Ubuntu 20.04 LTS, which remains in mainline support until April 2023. As with previous releases, NVIDIA provides access to a range of pre-trained models — more than 100 architectures, plus over 25 task-based models — with training and customization supported via the TAO Toolkit on an external workstation or pay-to-play cloud compute platform.

The idea, NVIDIA explains, is to develop applications on the compact box alongside an existing workstation then deploy them on yet-to-be-released AGX Orin systems-on-module. As with previous Jetson developer kits, there’s no official support for direct development and training on-device — though you could certainly try, with the system booting as it does into a functional Ubuntu desktop environment.

Performance

NVIDIA’s previous Jetson modules have been gently criticized for weaker-than-anticipated CPU performance — which, in the company’s defense, has never been the focus of the boards. The AGX Orin, then, comes as a welcome surprise: its 12 Arm Cortex-A78AE cores, running at up to 2.2GHz in the highest power envelope, are a major upgrade over the eight cores of its predecessor — approaching, but not quite equaling, the performance of an eight-core 16-thread AMD Ryzen 2700X desktop processor at a considerably lower power draw.

On the topic of power draw, the new Ampere GPU in the AGX Orin definitely lives up to its name. The Jetson AGX Xavier could be configured for a 10W to a 30W target power envelope, disabling various cores to reduce power requirements as you walked down to the bare minimum. The AGX Orin, by contrast, has 15W as its absolute minimal power envelope — and hikes the upper end to 60W, twice that of its predecessor.

As with NVIDIA’s previous Jetson kits, the claimed power envelopes are measured purely at the system-on-chip — and are noticeably conservative as a result. The company’s claimed 60W draw in “MAXN” mode, which activates all CPU and GPU cores and removes any clock restrictions in place, is the widest of the mark at 98W when measured at the wall — a measurement which, in NVIDIA's defense, includes the inefficiency of the supplied USB power brick and overhead for the carrier board. Even at idle the Jetson AGX Orin is a hungry beast, drawing 22W in MAXN mode and over 19W even when configured for the lowest “15W” setting — though NVIDIA says this will be resolved in a future software update, enabling currently-disabled power saving features to reduce idle draw to under 5W.

All that power has to go somewhere, and that’s into a heatsink that forms part of the bundled and pre-installed case. While clearly an evolution of the one included with the AGX Xavier kit, the AGX Orin case is a big upgrade. The full-size PCI Express slot is now protected via a removable cover, albeit one tethered to the rest of the case by cables for a pair of adhesive antennas stuck on the inside, and there’s a clearer path for the on-board fan to pull air in and push it over the heatsink. The fan itself is speed-controlled according to temperature, and reasonably quiet for the amount of air it pushes at around 50dBA measured at arm’s length.

Hybrid module

Another thing developers need to keep in mind is that the system-on-module driving the Developer’s Kit doesn’t actually exist — at least, not as a stock-keeping unit you could slot into a finished design. There are two models of AGX Orin modules so-far confirmed, both launching later this year: a version with 32GB of RAM, which has eight CPU cores, 1,792 CUDA cores, 56 Tensor cores, and lower maximum GPU and NVDLA frequencies; and a 64GB model that has 12 CPU cores, 2,048 CUDA cores, and 64 Tensor cores plus higher maximum clock speeds.

The module at the heart of the Developer’s Kit, meanwhile, is a hybrid of the two. Its specifications match the higher-end 64GB model in almost all respects — a claimed 275 TOPS of sparse INT8 compute to the lower-end model’s 200 TOPS, all three four-core CPU clusters present and correct, full clock speeds for CPU and NVDLA — except for having 32GB of RAM in place. For those building projects that will fit in 32GB of memory, it offers a handy speed boost — but one which could provide a false impression of the performance you’ll get out of the true 32GB module once your project is deployed.

A big feature of the new Ampere GPU architecture is its support for sparse networks, offering a big speed bump where applicable. It’s also key to NVIDIA’s claims of 275 TOPS compute performance: While, yes, the AGX Orin can hit 275 TOPS of INT8 compute across all its cores and accelerators, it can only do so on sparse networks; on a standard dense network, as compatible with the previous-generation AGX Xavier, its throughput is lower at 138 TOPS — still an impressive uplift on the 32 TOPS of its predecessor, of course.

Generational gains and losses

Elsewhere, there are a few downgrades. There's only a single video output and H.265 video encode capabilities are roughly half those of the AGX Xavier, though there’s now support for hardware AV1 encoding in place of VP9; hardware decode is similarly halved, supporting a maximum of one 8k30 stream to the AGX Xavier’s two, though with AV1 added alongside VP9, H.265, and H.264. The 16 lanes of MIPI CSI-2 now support only 16 virtual channels to the AGX Xavier’s 36, though the C-PHY has been upgraded to 2.0 with a boosted throughput of 164Gbps to its predecessor’s 62Gbps.

These come alongside some major upgrades, though. The move to LPDDR5 offers a measurable boost to memory bandwidth, the new CPU cores are considerably improved and there are more of them, the GPU is easily more than three times as fast for FP32 workloads, there are four 10-gig-Ethernet connections on the module with one brought out to RJ45 on the carrier board, and the on-board eMMC has been doubled to 64GB — though that’s still a little small for comfort, and an M.2 NVMe drive should be considered a must-have accessory when picking up the kit.

Which brings us to the kit’s biggest drawback: the price. At $1,999, the AGX Orin Developer’s Kit is a full $700 more expensive than the AGX Xavier was when purchased through the NVIDIA Developer Program at launch. It’s also not an all-in-one kit: as well as NVMe storage, buyers will have to budget for a traditional desktop or laptop and NVIDIA GPU on which to run training — or sign up to a cloud computing platform for the same reason — as there’s no official way to run training on the AGX Orin directly, despite its impressive specifications.

The Jetson AGX Orin Developer’s Kit is now available to order via the NVIDIA website at $1,999; the Jetson AGX Xavier Developer’s Kit was formerly available as low as $649, but ongoing industry-wide component shortages mean it’s been out of stock for some time.