Overview of Nvidia Jetson Orin Nano Development Kit

Introduction

Nvidia recently launched the Jetson Orin Nano super development kit, an improvement on last year's Jetson Orin Nano development kit. This document examines the differences between the two kits, demonstrates running large language models on a single Jetson Orin Nano device, and explores cluster configurations for running larger models.

Core Differences

• Hardware: No changes in hardware from the previous Jetson Orin Nano development kit.
• Software: Significant improvements leading to performance boosts, achieved without new hardware.

Key Enhancements

• Price Reduction: From $500 to $250, making the kit more accessible.
• Performance Increase:
  • AI performance increased from 40 TOPS to 67 TOPS.
  • Memory bandwidth increased from 68 GB/s to 102 GB/s, providing a 1.7x performance boost for generative AI models.

Technical Specifications

• CPU: 6-core ARM Cortex A78AE running at 1.7 GHz (overclocked from 1.5 GHz).
• GPU: 1024-core Ampere architecture GPU, clocked at 1.2 GHz (up from 625 MHz).
• Memory Bandwidth: Now 102 GB/s, aiding in substantial performance increases.
• Power Modes: New Max-N mode offering 25 watts, in addition to previous modes.

Demonstrations

Running Models Locally

• Demonstrates running Llama 3.2 (3 billion parameters) and Gemma 2 (2 billion parameters) large language models locally.
  • Llama 3.2 Token Rate: 20.5 tokens/sec
  • Gemma 2 Token Rate: 21 tokens/sec

Cluster Setup for Larger Models

• Models larger than 8 GB can be run across multiple Jetson devices using a cluster setup.
• Demonstration with Llama CPP tool to split a 9.2 GB model (Gemma 29 billion parameters) over two devices, indicating the feasibility of running larger models despite reduced speed due to network constraints.

Conclusion

• Nvidia's update is primarily software-driven, offering substantial performance improvements without new hardware.
• The price cut and performance boost significantly enhance the device's appeal for running advanced AI models at the edge.
• Open potential for larger model deployment using clusters of Jetson Orin Nano kits.

Final Thoughts

The Jetson Orin Nano super development kit is a powerful tool for AI model deployment, whether for individual developers or scaled projects needing clustered configurations. The upgrade strategy represents a user-friendly approach to leveraging existing hardware with smarter software enhancements.