AI On the Edge, Raspberry Pi

AI on the Edge LESSON 33: Tracking Faces with Pan Tilt Camera in OpenCV on Pi 5

In this video lesson we find faces and eyes, and then track the face using the Pan Tilt camera on the AI Fusion Lab Kit. We use Haar Cascades to find the faces. Then we have a small green ‘On’ button on the frame, and a small red ‘Off’ button. We are able to do this at 15 FPS. The code we developed in the video is available below.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 32: Facial Recognition and Eye Tracking in OpenCV

Hey guys, Paul McWhorter here from TopTechBoy.com. Welcome back to our AI on the Edge series. If you’ve been following along, you already know how to pull high-frame-rate video off your Raspberry Pi 5 using the new picamera2 library, and you know how to use OpenCV to hunt down faces in a crowded frame.

But today, we are taking things a massive step forward. We aren’t just looking for faces anymore—we are looking inside the face to track the eyes.

This lesson highlights one of the most vital concepts in all of computer vision: The Region of Interest (ROI). If you try to scan an entire 1280×720 frame for tiny features like eyes, your frame rate will absolutely tank. Instead, we are going to act like real engineers. We will use a cascading logic approach: find the face first, isolate that exact box, and search only inside that small window for the eyes.

Go ahead and pour yourself a nice, cold glass of iced coffee or a hot cup of black coffee, get your code ready, and let’s break down exactly how this program works.

This is the code we developed in the video:

Code Architecture & Codex Breakdown

Since you already have the script loaded up in your IDE, let’s dissect the critical logic gates that make this tracking script fast and accurate.

1. Setting Up the High-Performance Pipeline

We configure the Picamera2 frontend to grab a crisp RGB888 array at a resolution of 1280×720 targeting 60 FPS. By using .capture_array(), we bypass slow formatting overhead and feed raw pixel data directly into OpenCV. Because the camera orientation might be flipped depending on your desktop mounting rig, we use cv2.flip(frame, -1) to keep the spatial coordinates intuitive.

2. The Cascading Filter Matrix

Notice how we initialize two distinct classifiers using pre-trained Haar Cascades:

  • haarcascade_frontalface_default.xml (To grab the macro features of the face)

  • haarcascade_eye.xml (To grab the micro features of the eyes)

We pass a minSize parameter of 100×100 pixels for the face detector. Why? Because we don’t care about background noise or tiny false positives across the room. We want to find you, sitting right in front of the workstation.

3. The Magic of the Region of Interest (ROI)

This is where the real engineering happens. Look closely at this inner loop:

Instead of passing the massive gray frame to the eye finder, we slice the array: gray[y:y+h, x:x+w]. This isolates a tiny sub-matrix containing nothing but your face. The search area drops exponentially, keeping our frame rates close to maximum velocity.

4. Re-Mapping Local to Global Coordinates

When the eye detector finds a match inside the sliced face frame, it returns local coordinates (i, j, w, h) relative to the top-left corner of that face box, not the whole screen. If you tried to draw a rectangle directly at (i, j), your eye boxes would be floating erratically in the top-left corner of your monitor!

To fix this spatial offset, we map them back to global coordinate space by adding the face’s original offsets:

  • Global X Position: x + i

  • Global Y Position: y + j

General Knowledge: How Haar Cascades and ROIs Work Under the Hood

Now that you understand the mechanics of the script, let’s dive into the fundamental computer vision theory that makes legacy Edge AI tracking so efficient.

The Viola-Jones Framework

Haar Cascade classifiers are based on the Viola-Jones object detection framework. Instead of using massive, compute-heavy deep learning neural networks that require powerful discrete GPUs, Haar Cascades utilize simple, binary pixel-intensity features called Haar-like features.

These features act like digital templates looking for specific shifts in brightness:

  • Edge Features: Detects boundaries where a dark zone transitions into a light zone (like the bridge of your nose versus your cheek).

  • Line Features: Useful for identifying long, horizontal elements like eyebrows or the line of the mouth.

  • Center-Surround Features: Excellent for finding eyes, where the dark pupil is surrounded by lighter skin and sclera.

Why Slicing the Array Saves Your Processor

Every time you invoke .detectMultiScale(), OpenCV has to pass a sliding window across the image matrix at multiple scales, performing thousands of additions and subtractions per frame.

Mathematically, if an entire frame has a pixel area, scanning it scales linearly with that total area. By filtering for the face first and establishing a tight Region of Interest (ROI), you reduce the eye tracking search space down to a fractional area.

On resource-constrained hardware like an edge microcontroller or a single-board computer, isolating the matrix dimensions before calling nested lookups is the difference between a sluggish, unusable slideshow and a silky smooth tracking experience.

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 31: Facial Recognition in OpenCV Using Haarcascades

Hey everyone, this is Paul McWhorter from toptechboy.com. Welcome back to our AI on the Edge tutorial series. If you’ve made it this short distance through the course, you are doing fantastic.

Today, we are stepping away from simply pulling a clean, high-frame-rate video stream off our hardware, and we are finally going to start doing some real Computer Vision. We are going to teach our machine how to look at an image, process it, and actually locate human faces in real time.

Go ahead and grab yourself a hot cup of coffee. Turn off your cell phone, close your other browser tabs, and let’s get ready to write some serious engineering code.

The Core Concept: What is a Haar Cascade?

Up until now, we’ve been focused on the plumbing—getting the camera configured, setting up the dimensions, and flipping the frames so they look right on our screens. Today, we introduce the Haar Cascade Classifier.

Think of a Haar Cascade as an incredibly smart, pre-trained statistical filter. Decades ago, researchers realized that human faces share universal geometric patterns of light and dark. For example:

  • The bridge of your nose is almost always brighter than the shadows on either side of it.

  • Your eye sockets are consistently darker than your forehead or your cheekbones.

OpenCV comes pre-packaged with these mathematical descriptions of a face. The algorithm works by taking a tiny “sliding window” and dragging it across your video frame pixel by pixel, looking for those specific arrangements of light and dark gradients. When it finds a cluster that matches the math, it flags it as a face.

Why Grayscale Matters in Machine Learning

If you look closely at our pipeline logic for today, the absolute first thing we do after capturing a raw frame from our camera is strip away all the color and convert the image to grayscale.

Why do we do this? Color is a computational luxury we cannot afford when doing real-time edge processing. To a computer, a color image consists of three separate channels: Red, Green, and Blue. That means for a standard resolution, the processor has to crunch three times the data.

Because Haar Cascades only care about the contrast—the relationship between light and dark areas—the actual color of your skin, your hair, or your shirt is completely irrelevant. By dropping the frame down to a single grayscale intensity channel, we cut our processor’s workload by 66% instantly. This is how we keep our edge hardware running lightning-fast without lagging the video feed.

Tuning the Detection Engine

When we tell OpenCV to look for faces using the detectMultiScale function, we pass three critical parameters that you need to master. If you don’t tune these right, your program will either miss faces entirely or start thinking your coffee mug or a pattern on the wall is a human being.

  • scaleFactor=1.1: A single Haar Cascade model is trained at a fixed size. But a face could be right next to the lens (huge) or all the way across the room (tiny). The scaleFactor tells OpenCV to shrink the image by 10% on each pass, creating a “layer cake” of images at different scales so the sliding window can catch faces of any size.

  • minNeighbors=5: As the sliding window moves, it might find dozens of potential matches around your eyes and nose. This parameter states that a face must be detected in at least 5 overlapping bounding boxes before the system officially declares, “Yes, that is a human face.” Raising this number reduces false positives but makes the system less sensitive.

  • minSize=(60,60): This tells the algorithm to completely ignore any detected objects smaller than a 60×60 pixel block. This prevents the system from wasting CPU cycles trying to analyze tiny bits of background noise in the distance.

The Secret to a Smooth FPS Counter

You will notice a very specific mathematical formula used to calculate our Frames Per Second (FPS) in this lesson. If you simply calculate $1 / \Delta T$ every single frame, your FPS display will jitter wildly on screen, flashing unreadable numbers back and forth because individual frames might take a millisecond more or less to process.

To fix this, we implement a digital Low-Pass Filter.

Every loop, we take 95% of our previous FPS value and add only 5% of our current instantaneous speed measurement. This creates a beautiful, smooth running average that responds immediately to system slowdowns but remains steady and completely legible on the screen. It is an elegant engineering solution to a common UI headache.

Homework Assignment!

You didn’t think you were going to get out of here without homework, did you? No shortcuts in this classroom!

Now that your program can successfully locate faces and draw a bounding box around them, it’s time to take it to the next level. Your assignment is to modify this program to isolate and track only the single largest face in the frame.

If multiple people walk into the camera’s view, your script must evaluate the dimensions of the returned bounding boxes, determine which person is closest to the lens (the largest box), and draw a bright green rectangle around only that leader face, while ignoring everyone else.

Here is the code developed in today’s lesson.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 30: Tune Object Tracker with Mouse Selected ROI

AI on the Edge LESSON 30: Tune Object Tracker with Mouse Selected ROI

Welcome, Makers!

Well, hello there! It is absolutely fantastic to have you back. I’m Paul McWhorter, and today, we are taking a massive step forward in our AI on the Edge journey.

Up until now, we’ve been hard-coding our color thresholds (those pesky Lower Color and Upper Color values) to tell our camera what to look for. That’s fine for a science experiment, but it’s not exactly “smart,” is it? If the lighting changes, or if we want to track a different colored object, we have to go back into the code and manually edit those numbers.

Not anymore!

In today’s lesson, we are building a tool that lets us teach the AI. We’re going to use the mouse to draw a Region of Interest (ROI) right on our camera feed. The system will look at the pixels inside that box, calculate the average Hue, Saturation, and Value, and automatically set our tracking range for us.

This is the kind of professional-level functionality that turns a hobby project into a true, intelligent machine.

The Concept: From Hard-Coding to Dynamic Learning

The magic happens in our mouseAction function. Instead of just reading pixel values, we are now implementing a “click-and-drag” system:

  1. Click and Hold: We capture the startX and startY coordinates.

  2. Drag: We draw a rectangle in real-time so we can see exactly what area we are selecting.

  3. Release: We take that specific slice of the image, convert it to the HSV color space, and use the cv2.mean() function to find the average color properties.

  4. Auto-Tune: We set our LC (Lower Color) and UC (Upper Color) based on that average.

By doing this, the system learns what “object” we want to track on the fly. It’s elegant, it’s powerful, and it feels like real magic when you see those servos snap onto your target after a quick mouse drag.

What We’ve Accomplished

By the end of this lesson, you will have a system that:

  • Visually selects an object using the mouse.

  • Automatically calculates the optimal HSV thresholds for that specific object.

  • Updates the tracking behavior immediately without needing to stop or re-run the code.

  • Maintains that professional “Edge” feel, giving you real-time feedback on your FPS and mouse position data.

A Note on the “Edge”

Remember, we aren’t just running code; we are running on hardware. When we calculate the mean of the ROI, we are doing real image processing on the fly. You’ll notice the Composite and Mask windows updated immediately, giving you a visual confirmation that your “teacher” (you!) has successfully guided the “student” (the AI).

This is the power of working with OpenCV and the Raspberry Pi. You are building a system that observes, thinks, and reacts—all in real-time.

Get Ready to Build

Grab your Pi, make sure your servos are ready to go, and let’s get that camera calibrated. You’ve put in the work to get this far, and today is where all that effort starts to feel really rewarding.

I’m incredibly proud of how far you’ve come. Let’s dive in and start building!

Are you ready to see how accurately your Pi can “see” once you’ve given it the ability to learn from your selections?

In the video lesson we developed the following code.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 29: Improved Proportional Object Tracking with Pan Tilt Camera

AI on the Edge LESSON 29: Improved Proportional Object Tracking with Pan Tilt Camera

Hey everyone, Paul McWhorter here from TopTechBoy.com. Welcome back to our channel, where we learn to build real, intelligent systems on edge hardware. Go ahead and grab yourself a nice hot cup of coffee or a big glass of iced tea, because today we are going to completely revolutionize the way our robotic pan-tilt camera interacts with the physical world.

In Lesson 28, we successfully closed the loop. We got our camera to physically move and track an object using the error signal calculated from our OpenCV bounding box. It worked, but let’s be honest with ourselves: it was clunky. It was a crude, incremental system that moved the camera by exactly one lazy degree at a time, regardless of whether the target was right next to the crosshairs or flying across the room. It was jerky, it hunted back and forth, and it just wasn’t elegant old-school engineering.

Today, we are throwing away that clunky incremental logic and replacing it with something beautiful: Proportional Control.

The Problem with Lazy Incremental Steps

Before we fix our control loop, we need to understand exactly why our previous system struggled. In our last script, we used conditional statements to see if the error was positive or negative, and then adjusted our angles by a fixed step of 1 or -1.

This created two major engineering flaws:

  • Lagging on Large Errors: If you suddenly jerked the object 400 pixels away from the center, the camera would take forever to catch up because it could only step at a constant speed of one degree per loop iteration.

  • Hunting and Jitter on Small Errors: When the object finally got close to the center, the camera would overshoot by a full degree, trip the opposite condition, and step back. It would constantly “hunt” back and forth across the target, buzzing your hardware to pieces.

The Elegance of Proportional Control

In real-world automation, we don’t use rigid, conditional step-programming to move hardware. We use mathematics. We want the camera’s reaction to be completely proportional to the size of the mistake it is trying to correct.

If the object is a massive distance away from the center crosshairs, we want the servo to take a massive, aggressive leap to catch up instantly. As the object gets closer and closer to the center, we want the camera to automatically slow down and gently glide into place. When the error drops to zero, the physical adjustment should naturally drop to zero.

The magic of this approach is that it allows us to completely eliminate the bulky conditional statements and artificial deadbands we wrote last time. The algebra naturally handles the direction and magnitude of the movement.

Breaking Down the Math and Logic

To achieve this fluid motion, we take our raw error signal—the distance in pixels between our frame center and the object center—and apply a scaling factor, known in control theory as Gain.

In this updated system design, we take our error and divide it down. Specifically, we divide the pixel error by 50, and then split that in half by dividing by 2. Mathematically, this means we are scaling our pixel error down by a factor of 100.

  • If your object is 300 pixels off-center, the math calculates an instantaneous adjustment of 3 degrees, quickly snapping the camera toward the target.

  • If the object is only 10 pixels off-center, the adjustment becomes a tiny fraction of a degree (0.1), smoothly stabilizing the camera track.

Precision Tracking with Floating-Point Variables

Because we are dividing our pixel error down by 100, our angular adjustments will almost always be fractional decimals rather than clean integers. If we tried to store these angles as standard integers, our program would truncate those decimals, completely throwing away our precise micro-adjustments and causing the camera to stall out.

To make this system work perfectly, we maintain our accumulation variables as high-precision floating-point numbers. The script constantly adds and subtracts these fractional updates over time behind the scenes. We only cast the final calculated angle to a clean, rounded integer at the absolute last microsecond right as we pass the position command to the physical servo motors.

Visual Tuning and Smooth Performance

You will notice a massive visual upgrade when running this refined loop. To match our new high-precision math, we tighten up our tracking reticle overlay, shrinking our target circle down from a radius of 40 to a crisp 30 pixels. We also change our dynamic bounding box to a bright, vibrant yellow to make our tracking visually pop on screen.

When you fire up this loop and wave your object around, you will see a night-and-day difference compared to last week. The lazy, robotic stutter is completely gone. The pan-tilt mount tracks with an organic, fluid motion, actively accelerating and decelerating to mirror your movements perfectly.

 

Making The World a Better Place One High Tech Project at a Time. Enjoy!