2025-11-19: Edge AI with Jetson Orin Nano

By -

Building AI-powered applications at the edge has never been easier. The Jetson Orin Nano Super Developer Kit is a budget-friendly (Amazon) way to get started. LLMs, VLMs & VITs are some applications we run. It delivers up to 67 TOPS of AI performance and memory bandwidth of 102 GB/s.

Let's get started.

In addition to the Jetson Orin Nano Super Developer Kit, we need the following:

  1. An NVIDIA account.
  2. A workstation (host machine),
    • Running Ubuntu 22.04 or 20.04
      [Run "lsb_release -a" command to check ubuntu version]
    • Minimum of free 100GB disk space in OS drive
    • Minimum of 8GB ram
    • USB 3.1 port
    • Good internet connection
  3. MicroSD card (minimum 64GB) or NVMe PCIe SSD (minimum 128GB)
    Here we have 3 options to configure the SSD and SD card.
    • Boot from SSD (ditch microSD card) ← Let's go with this method
    • Boot from microSD card (ditch SSD)
    • Boot from microSD card and keep SSD as external storage

      Why go for NVMe SSD? Faster (10X better performance over SD card), reliable, large capacity.

Steps

  1. Install SDK Manager on host machine by executing following commands in a terminal.

    For Ubuntu 22.04
    wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb

    For Ubuntu 20.04
    wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.1-1_all.deb

    And below commands: sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt-get update sudo apt-get -y install sdkmanager

  2. Launch SDK Manager with the following command:
    sdkmanager

  3. Login to your NVIDIA account.

  4. Once SDK Manager is launched, connect your Jetson developer kit to your Ubuntu PC and power it on in Forced Recovery mode with the steps below:
    • Install the SSD card.
    • Shortpin 9andpin 10ofJ14header located below the Jetson module using a jumper pin.
    • Power it on.
    • Connect it to your Ubuntu PC with a USB cable using USB-C port on Jetson Nano.

  5. SDK manager steps
    1. Just after power it on the Jetson will be detected and in prompt, select "Jetson Orin Nano [8GB developer kit version]" and hit "OK".

      If the Jetson was not automatically detected, run "lsusb" command to check if it's listed, otherwise check your USB-C cable connection.

    2. Uncheck "Host Machine" if you don't need to install NVIDIA components and development tools on your host computer.

    3. Check "Jetson Linux" component and check other components as well if you need them right away. (Or you can always install them later.)

    4. Check "I accept the terms and conditions of the license agreements" and Click "Continue" button.
    5. It will prompt for thesudocommand password of the host machine.

    6. Once images are ready, SDK it will open the prompt for flashing. Select Storage Device "NVMe". If you have selected anything other than "Jetson Linux" component in step 5.3 select OEM configuration "Pre-Config", otherwise you have to configure OS manually and connect the Jetson Nano again in Forced Recovery mode to host machine to continue installation of SDK/Runtime components.

    7. Flash progress.

    8. After flashing, it will prompt below screen to start install SDK/Runtime components and click "Install".

    9. SDK/Runtime installation progress.

    10. After the installation finishes, click the "FINISH" button.

  6. Boot and start
    1. Power off.
    2. Remove jumper.
    3. Connect display, mouse, keyboard.
    4. Power on.


Congratulations! Now you are ready to develop on Jetson.

Well not just yet. Let's do the configurations below as well:

  • Switch to MAXN SUPER mode
    Default power mode on Jetson Orin Nano Developer Kit is 25W. Switch to the MAXN SUPER mode and enable maximum performance.

  • Disable the desktop GUI
    This will free up extra memory that the window manager and desktop uses (around ~800MB)
    We can disable the desktop temporarily, by running below commands in the console.
    $ sudo init 3 # stop the desktop # log your user back into the console $ sudo init 5 # restart the desktop # log your user back into GUI
    Now there are 2 ways to work.
    1. Use console with monitor


    2. Use Jetson remotely through SSH
      1. Connect the Jetson to any PC with a USB cable, through the USB-C port.
      2. You should see a "L4T-README" drive automatically gets mounted on your PC.
      3. Open a terminal on your PC and SSH into the Jetson using the following command:
        ssh ${JETSON_USERNAME}@192.168.55.1

        Now we don't need a separate monitor/keyboard/mouse. And you can configure above ssh command with your favorite IDE.


      4. Before unplugging the DC cable, remember to power off the system with the following command:
        sudo poweroff
  • Install Miniconda and Create Python Environments
    We need isolated python environments for different projects. For that we'll need to install Miniconda on Jetson Orin and create a Python environment. Run the commands below and you are good to go.

    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh chmod +x Miniconda3-latest-Linux-aarch64.sh ./Miniconda3-latest-Linux-aarch64.sh conda update conda conda create -n test python=3.10 conda activate test

Since we have come already this far, let's run small LLM using Ollama for a test drive.

To run LLMs locally, install and run Ollama:

# install CUDA-supporting Ollama curl -fsSL https://ollama.com/install.sh | sh # run LLaMA 3.2 locally with 3B parameters ollama run llama3.2:3b



In this post, we learned how to set up and configure the Jetson Orin Nano for AI development. With this setup ready, you can start exploring and developing exciting Generative AI applications and models right on your own device.


Pasindu Thenahandi
PhD Student, Computer Science
Old Dominion University
Norfolk, VA 23529
Email: pasindu@cs.odu.edu

Comments

Post a Comment