NVIDIA, NVIDIA Jetson Orin Nano

Running Headless on the NVIDIA Jetson Orin Nano on Jetpack 7.2: Run Big Local LLM’s Like a Boss

Hey guys, Paul McWhorter here from toptechboy.com. Today, we are going to look at how to stop fighting your hardware and start running local AI like an absolute boss.

The NVIDIA Jetson Orin Nano is an absolute masterpiece of edge-compute hardware. But it has one major design constraint that catches almost every beginner off guard: Unified Memory. On the Orin, your CPU and your GPU share the exact same physical pool of 8GB of LPDDR5 RAM. When you boot into that pretty Ubuntu GNOME desktop, the system instantly steals over 1.5 GB of your precious VRAM just to draw a GUI you aren’t even looking at while your code is running.

In this lesson, we are going to reclaim that stolen memory, optimize our storage, and run a massive 8-billion parameter model (LLaMA 3.1 8B) smoothly on the Orin Nano by learning how to properly run a clean, headless configuration.

Step 1: Find Your Jetson’s IP Address

Before we turn off the monitor, we need to know how to talk to the Orin over the network. If you don’t know its IP address, you can’t SSH in once the screen goes dark. While you are still in the graphical terminal, run this command:

Look under your active connection interface (usually eth0 for Ethernet or wlan0 for Wi-Fi) for the inet address. It will look something like 192.168.1.15. Write this down! You will need it to remote in later.

Step 2: Disable and Remove the Default Swap File

By default, JetPack configures a slow, disk-based swap file on your NVMe drive. While swap space is great for general computing, it is an absolute performance killer for LLMs. If your model spillover starts paging to a disk-based swap file, your tokens-per-second will drop to a crawl, and the high-frequency writes will prematurely wear out your SSD.

We want our models running purely in ultra-fast LPDDR5 RAM. Let’s cleanly turn off and remove the swap file:

Step 3: Configure a Clean Boot into the Terminal

Many people will tell you to run sudo systemctl isolate multi-user.target to turn off the GUI. Do not do this! That command aggressively tears down active background services (including Ollama, network managers, and local development scripts) because it forces a state isolation.

Instead, we want to tell the Orin’s bootloader to cleanly start up in command-line mode from a fresh boot. This allows all your network drivers, background scripts, and Ollama to initialize perfectly without a display manager eating your memory:

Once you run this, restart your Orin to let the changes take effect cleanly:

How to Boot Back to the GUI (If Needed)

We are developers, which means we want to write and debug our scripts comfortably under the graphical desktop, and then deploy them headlessly. If you ever need to turn your monitor back on and return to the GNOME desktop, simply run this command over SSH:

Followed by a quick reboot (sudo reboot), and your desktop interface will return exactly as it was.

Step 5: The Test โ€” Running LLaMA 3.1 8B in the GUI

To prove why this matters, let’s look at what happens when you try to force a large model to run while your monitor is plugged in and the graphical desktop is active. Open your terminal in the GUI and run:

The Result: The model will either completely crash with an “Out of Memory” (OOM) error, or it will run painfully slow, chugging out less than 2 tokens per second.

The “Why”: Where Did Your Memory Go?

An 8-billion parameter model quantized to 4-bits requires roughly 4.7 GB of static memory just to fit its weights. When you add the Context Window (KV Cache), that memory requirement quickly balloons to over 5.5 GB.

Here is exactly how your 8GB Orin Nano’s memory is divided when you run a GUI:

System State Memory Allocation (Approximate)
OS Kernel & System Daemons ~1.2 GB
GNOME Desktop GUI (Monitor Active) ~1.6 GB
Available VRAM for AI ~5.2 GB (Not enough for 8B models + Context!)

Because the GUI steals 1.6 GB, your available memory drops below the critical threshold required to run LLaMA 3.1 8B. The moment your context grows, the system runs out of room, hits a bottleneck, or crashes.

Step 6: Reclaiming the Hardware (Headless Memory Profile)

Now let’s look at the memory profile when we boot the Orin Nano cleanly into the terminal without GDM3 starting up. If you SSH in and run free -h or check jtop, this is what you get:

System State Memory Allocation (Approximate)
OS Kernel & System Daemons ~1.2 GB
GNOME Desktop GUI 0.0 GB (COMPLETELY RECLAIMED!)
Available VRAM for AI ~6.8 GB (Plenty of headroom for 8B models!)

By going headless, we instantly reclaimed **1.6 GB of ultra-fast VRAM**. That is the difference between night and day when deploying edge AI models.

Step 7: Connect from Windows PowerShell

Now that your Orin is booted headlessly, unplug the monitor, keyboard, and mouse. Walk back to your main Windows development machine, open up **PowerShell**, and SSH directly into the Orin over your local network using the IP address you saved in Step 1:

(Be sure to replace “pjm” with your actual Orin username and use your specific IP address!)

Step 8: Run LLaMA 3.1 8B Like a Boss

With your GUI safely dead and your memory completely optimized, run the exact same model command inside your PowerShell session:

The Payoff: Because the system now has a massive 6.8 GB of free, continuous VRAM, the model loads entirely into the Orin’s hardware engines. You will see prompt evaluations complete instantly, and the text will output at an extremely usable speed without a single memory warning or system hiccup.

That is how you cleanly manage your hardware resources, develop efficiently, and run large local LLMs on the edge like an absolute boss.

If you enjoyed this write-up, leave a comment below, subscribe to the channel, and I will see you guys in the next lesson!

๐ŸŽ“ Homework: Show Your Work!

Alright guys, no excuses! If you want to truly master this hardware, you cannot just sit there and watch me do itโ€”you have to get your hands dirty. For your homework today, I want to see you running your own LLaMA 3.1 8B model headlessly on your Orin Nano. Show what tokens per second you are getting on this big modal. Create your own favorite query to show how well the model works. Show me that terminal proof and the memory savings!

Here is the plan:

  • Record a video of your setup successfully running the model headlessly.
  • Upload your video to YouTube.
  • In the description of your YouTube video, you must include a link back to this main tutorial video at the very top of your description.
  • Post a link to your homework video in the comments section on the video above, running your models like a boss.

Now, get to work! I am looking forward to seeing what you guys build.

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 33: Tracking Faces with Pan Tilt Camera in OpenCV on Pi 5

Hey guys, Paul McWhorter here from toptechboy.com. In today’s lesson, we are taking our “AI on the Edge” skills to the next level. We aren’t just detecting faces anymore; we are going to make our camera system react to them.

We are integrating our computer vision logic with physical hardware to create a pan-tilt tracking system. We’ll use the Raspberry Pi 5 to run high-speed inference, detect a face, calculate exactly how far that face has drifted from the center of our frame, and then command our servos to follow it in real-time. It is one thing to see a box draw around a face; it is a completely different level of “cool” when the camera actually turns to look at you.

The Engineering Concept: The Error Loop

In robotics, this is a classic control problem. We have a Target (the center of the face) and a Setpoint (the center of our camera frame). The difference between these two points is our Error.

  • xError: How far left or right the face is from the center.
  • yError: How far up or down the face is from the center.

By taking that error and dividing it by a “gain” constant (in our case, 50/2), we can smoothly adjust the servo angles. If we don’t divide by a constant, the camera will snap aggressively to the target or overshoot it. This simple division creates a “proportional” response that keeps our tracking smooth and precise.

What to Focus On

Make sure you have your picamera2 and fusion_hat libraries updated and configured before you dive in. The key to this lesson isn’t just getting the servos to moveโ€”it’s understanding how to bridge the gap between the coordinates returned by OpenCV and the angle coordinates required by your servo library.

Pay close attention to how we calculate the center of the frame and the center of the detected face box. Once you understand that math, you can use this same logic to track anything: faces, colored objects, or even specific shapes!

Homework Assignment: Show Your Work!

Alright guys, no excuses! If you want to truly master this hardware, you cannot just sit there and watch me do itโ€”you have to write the code and run it yourself.

Your Task: Get this tracking system working. Once you have it tracking a face, I want you to experiment with that “Gain” factor (the 50/2 part). Try increasing it and decreasing it. What happens to the tracking quality? Is it smoother? Does it jitter?

Record a video of your camera following your face around the room, upload it to YouTube, and link back to this video at the top of your page. Post a link to your homework in the comments section below so I can see you running your code like a boss.

 

The code we developed in the video is available below.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 32: Facial Recognition and Eye Tracking in OpenCV

Hey guys, Paul McWhorter here from TopTechBoy.com. Welcome back to our AI on the Edge series. If youโ€™ve been following along, you already know how to pull high-frame-rate video off your Raspberry Pi 5 using the new picamera2 library, and you know how to use OpenCV to hunt down faces in a crowded frame.

But today, we are taking things a massive step forward. We aren’t just looking for faces anymoreโ€”we are looking inside the face to track the eyes.

This lesson highlights one of the most vital concepts in all of computer vision: The Region of Interest (ROI). If you try to scan an entire 1280×720 frame for tiny features like eyes, your frame rate will absolutely tank. Instead, we are going to act like real engineers. We will use a cascading logic approach: find the face first, isolate that exact box, and search only inside that small window for the eyes.

Go ahead and pour yourself a nice, cold glass of iced coffee or a hot cup of black coffee, get your code ready, and let’s break down exactly how this program works.

This is the code we developed in the video:

Code Architecture & Codex Breakdown

Since you already have the script loaded up in your IDE, let’s dissect the critical logic gates that make this tracking script fast and accurate.

1. Setting Up the High-Performance Pipeline

We configure the Picamera2 frontend to grab a crisp RGB888 array at a resolution of 1280×720 targeting 60 FPS. By using .capture_array(), we bypass slow formatting overhead and feed raw pixel data directly into OpenCV. Because the camera orientation might be flipped depending on your desktop mounting rig, we use cv2.flip(frame, -1) to keep the spatial coordinates intuitive.

2. The Cascading Filter Matrix

Notice how we initialize two distinct classifiers using pre-trained Haar Cascades:

  • haarcascade_frontalface_default.xml (To grab the macro features of the face)

  • haarcascade_eye.xml (To grab the micro features of the eyes)

We pass a minSize parameter of 100×100 pixels for the face detector. Why? Because we don’t care about background noise or tiny false positives across the room. We want to find you, sitting right in front of the workstation.

3. The Magic of the Region of Interest (ROI)

This is where the real engineering happens. Look closely at this inner loop:

Instead of passing the massive gray frame to the eye finder, we slice the array: gray[y:y+h, x:x+w]. This isolates a tiny sub-matrix containing nothing but your face. The search area drops exponentially, keeping our frame rates close to maximum velocity.

4. Re-Mapping Local to Global Coordinates

When the eye detector finds a match inside the sliced face frame, it returns local coordinates (i, j, w, h) relative to the top-left corner of that face box, not the whole screen. If you tried to draw a rectangle directly at (i, j), your eye boxes would be floating erratically in the top-left corner of your monitor!

To fix this spatial offset, we map them back to global coordinate space by adding the face’s original offsets:

  • Global X Position: x + i

  • Global Y Position: y + j

General Knowledge: How Haar Cascades and ROIs Work Under the Hood

Now that you understand the mechanics of the script, letโ€™s dive into the fundamental computer vision theory that makes legacy Edge AI tracking so efficient.

The Viola-Jones Framework

Haar Cascade classifiers are based on the Viola-Jones object detection framework. Instead of using massive, compute-heavy deep learning neural networks that require powerful discrete GPUs, Haar Cascades utilize simple, binary pixel-intensity features called Haar-like features.

These features act like digital templates looking for specific shifts in brightness:

  • Edge Features: Detects boundaries where a dark zone transitions into a light zone (like the bridge of your nose versus your cheek).

  • Line Features: Useful for identifying long, horizontal elements like eyebrows or the line of the mouth.

  • Center-Surround Features: Excellent for finding eyes, where the dark pupil is surrounded by lighter skin and sclera.

Why Slicing the Array Saves Your Processor

Every time you invoke .detectMultiScale(), OpenCV has to pass a sliding window across the image matrix at multiple scales, performing thousands of additions and subtractions per frame.

Mathematically, if an entire frame has a pixel area, scanning it scales linearly with that total area. By filtering for the face first and establishing a tight Region of Interest (ROI), you reduce the eye tracking search space down to a fractional area.

On resource-constrained hardware like an edge microcontroller or a single-board computer, isolating the matrix dimensions before calling nested lookups is the difference between a sluggish, unusable slideshow and a silky smooth tracking experience.

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 31: Facial Recognition in OpenCV Using Haarcascades

Hey everyone, this is Paul McWhorter from toptechboy.com. Welcome back to our AI on the Edge tutorial series. If you’ve made it this short distance through the course, you are doing fantastic.

Today, we are stepping away from simply pulling a clean, high-frame-rate video stream off our hardware, and we are finally going to start doing some real Computer Vision. We are going to teach our machine how to look at an image, process it, and actually locate human faces in real time.

Go ahead and grab yourself a hot cup of coffee. Turn off your cell phone, close your other browser tabs, and letโ€™s get ready to write some serious engineering code.

The Core Concept: What is a Haar Cascade?

Up until now, weโ€™ve been focused on the plumbingโ€”getting the camera configured, setting up the dimensions, and flipping the frames so they look right on our screens. Today, we introduce the Haar Cascade Classifier.

Think of a Haar Cascade as an incredibly smart, pre-trained statistical filter. Decades ago, researchers realized that human faces share universal geometric patterns of light and dark. For example:

  • The bridge of your nose is almost always brighter than the shadows on either side of it.

  • Your eye sockets are consistently darker than your forehead or your cheekbones.

OpenCV comes pre-packaged with these mathematical descriptions of a face. The algorithm works by taking a tiny “sliding window” and dragging it across your video frame pixel by pixel, looking for those specific arrangements of light and dark gradients. When it finds a cluster that matches the math, it flags it as a face.

Why Grayscale Matters in Machine Learning

If you look closely at our pipeline logic for today, the absolute first thing we do after capturing a raw frame from our camera is strip away all the color and convert the image to grayscale.

Why do we do this? Color is a computational luxury we cannot afford when doing real-time edge processing. To a computer, a color image consists of three separate channels: Red, Green, and Blue. That means for a standard resolution, the processor has to crunch three times the data.

Because Haar Cascades only care about the contrastโ€”the relationship between light and dark areasโ€”the actual color of your skin, your hair, or your shirt is completely irrelevant. By dropping the frame down to a single grayscale intensity channel, we cut our processor’s workload by 66% instantly. This is how we keep our edge hardware running lightning-fast without lagging the video feed.

Tuning the Detection Engine

When we tell OpenCV to look for faces using the detectMultiScale function, we pass three critical parameters that you need to master. If you don’t tune these right, your program will either miss faces entirely or start thinking your coffee mug or a pattern on the wall is a human being.

  • scaleFactor=1.1: A single Haar Cascade model is trained at a fixed size. But a face could be right next to the lens (huge) or all the way across the room (tiny). The scaleFactor tells OpenCV to shrink the image by 10% on each pass, creating a “layer cake” of images at different scales so the sliding window can catch faces of any size.

  • minNeighbors=5: As the sliding window moves, it might find dozens of potential matches around your eyes and nose. This parameter states that a face must be detected in at least 5 overlapping bounding boxes before the system officially declares, “Yes, that is a human face.” Raising this number reduces false positives but makes the system less sensitive.

  • minSize=(60,60): This tells the algorithm to completely ignore any detected objects smaller than a 60×60 pixel block. This prevents the system from wasting CPU cycles trying to analyze tiny bits of background noise in the distance.

The Secret to a Smooth FPS Counter

You will notice a very specific mathematical formula used to calculate our Frames Per Second (FPS) in this lesson. If you simply calculate $1 / \Delta T$ every single frame, your FPS display will jitter wildly on screen, flashing unreadable numbers back and forth because individual frames might take a millisecond more or less to process.

To fix this, we implement a digital Low-Pass Filter.

Every loop, we take 95% of our previous FPS value and add only 5% of our current instantaneous speed measurement. This creates a beautiful, smooth running average that responds immediately to system slowdowns but remains steady and completely legible on the screen. It is an elegant engineering solution to a common UI headache.

Homework Assignment!

You didn’t think you were going to get out of here without homework, did you? No shortcuts in this classroom!

Now that your program can successfully locate faces and draw a bounding box around them, it’s time to take it to the next level. Your assignment is to modify this program to isolate and track only the single largest face in the frame.

If multiple people walk into the camera’s view, your script must evaluate the dimensions of the returned bounding boxes, determine which person is closest to the lens (the largest box), and draw a bright green rectangle around only that leader face, while ignoring everyone else.

Here is the code developed in today’s lesson.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 30: Tune Object Tracker with Mouse Selected ROI

AI on the Edge LESSON 30: Tune Object Tracker with Mouse Selected ROI

Welcome, Makers!

Well, hello there! It is absolutely fantastic to have you back. Iโ€™m Paul McWhorter, and today, we are taking a massive step forward in our AI on the Edge journey.

Up until now, weโ€™ve been hard-coding our color thresholds (those pesky Lower Color and Upper Color values) to tell our camera what to look for. Thatโ€™s fine for a science experiment, but itโ€™s not exactly “smart,” is it? If the lighting changes, or if we want to track a different colored object, we have to go back into the code and manually edit those numbers.

Not anymore!

In today’s lesson, we are building a tool that lets us teach the AI. Weโ€™re going to use the mouse to draw a Region of Interest (ROI) right on our camera feed. The system will look at the pixels inside that box, calculate the average Hue, Saturation, and Value, and automatically set our tracking range for us.

This is the kind of professional-level functionality that turns a hobby project into a true, intelligent machine.

The Concept: From Hard-Coding to Dynamic Learning

The magic happens in our mouseAction function. Instead of just reading pixel values, we are now implementing a “click-and-drag” system:

  1. Click and Hold: We capture the startX and startY coordinates.

  2. Drag: We draw a rectangle in real-time so we can see exactly what area we are selecting.

  3. Release: We take that specific slice of the image, convert it to the HSV color space, and use the cv2.mean() function to find the average color properties.

  4. Auto-Tune: We set our LC (Lower Color) and UC (Upper Color) based on that average.

By doing this, the system learns what “object” we want to track on the fly. Itโ€™s elegant, itโ€™s powerful, and it feels like real magic when you see those servos snap onto your target after a quick mouse drag.

What Weโ€™ve Accomplished

By the end of this lesson, you will have a system that:

  • Visually selects an object using the mouse.

  • Automatically calculates the optimal HSV thresholds for that specific object.

  • Updates the tracking behavior immediately without needing to stop or re-run the code.

  • Maintains that professional “Edge” feel, giving you real-time feedback on your FPS and mouse position data.

A Note on the “Edge”

Remember, we aren’t just running code; we are running on hardware. When we calculate the mean of the ROI, we are doing real image processing on the fly. Youโ€™ll notice the Composite and Mask windows updated immediately, giving you a visual confirmation that your “teacher” (you!) has successfully guided the “student” (the AI).

This is the power of working with OpenCV and the Raspberry Pi. You are building a system that observes, thinks, and reactsโ€”all in real-time.

Get Ready to Build

Grab your Pi, make sure your servos are ready to go, and let’s get that camera calibrated. Youโ€™ve put in the work to get this far, and today is where all that effort starts to feel really rewarding.

Iโ€™m incredibly proud of how far youโ€™ve come. Letโ€™s dive in and start building!

Are you ready to see how accurately your Pi can “see” once you’ve given it the ability to learn from your selections?

In the video lesson we developed the following code.

 

Making The World a Better Place One High Tech Project at a Time. Enjoy!