AI On the Edge, Raspberry Pi

AI on the Edge LESSON 59: Object Detection with Active Scanning in MediaPipe with Pi 5

Hello everyone! Paul McWhorter here from toptechboy.com, and welcome back to our thrilling adventure into computer vision and edge AI. Today, we are dropping Lesson 59 of our AI on the Edge series.

In our previous sessions, we got our Raspberry Pi 5 looking at the world using MediaPipe object detection, we learned how to lock onto targets using pan-and-tilt servos, and we even brought in speech-to-text and text-to-speech capabilities via the Fusion Hat so our system could actually listen and talk back to us.

But there has been one nagging limitation: Static tracking. Up until now, if your object of interest wasn’t sitting right inside the camera’s initial field of view, your Pi just sat there blindly staring straight ahead, waiting for magic to happen.

Well, no more! Today, we are leveling up our system to feature Active Scanning.

The Motivation: From Passive Waiting to Active Hunting

Think about how human vision works. If you are looking for your coffee mug in a messy room, you don’t just stare at the wall and hope it walks past your nose. You move your head, you scan the environment, and your eyes sweep back and forth across different zones until you spot what you are looking for.

That is exactly what we are building in Lesson 59. We are combining three powerful pillars:

  1. Computer Vision (MediaPipe & OpenCV): Running blazing-fast object detection on the Raspberry Pi 5 using the PiCamera2 module.

  2. Robotic Actuation (Pan and Tilt Servos): Giving our camera a physical body with a dynamic range of motion.

  3. Conversational AI (Speech-to-Text & Text-to-Speech): Enabling voice command processing so you can simply tell your Pi what to find, while the system talks back to confirm its state.

How the Active Scanning System Works

In this lesson, our program operates across a few distinct operational modes managed seamlessly through threaded execution:

  • The Voice Listener Thread: Running independently in the background, this thread listens for your spoken commands using the speech-to-text engine. You can say things like “find”, “track”, “scan”, or name a specific object of interest (like “cup”, “person”, or “cell phone”). Saying “quit” gracefully terminates the application with a polite audio farewell.

  • Find Mode: The system rests or holds position while waiting for a target instruction or manual queue update.

  • Scan Mode: If the camera doesn’t immediately see your requested object of interest, it enters active scan mode. The pan servo sweeps smoothly back and forth across its horizontal range, stepping up or down the tilt angle after each pass. It systematically sweeps the room like a radar system.

  • Track Mode: The moment MediaPipe detects the target object with high confidence during a scan, the system instantly halts the scanning pattern, announces “Target Found” via text-to-speech, and switches over to active track mode. Using error correction based on the center of the bounding box versus the center of the screen, the servos continuously adjust to keep the target dead-centered in the frame.

What You Will Learn in the Video

In the accompanying video lesson, we break down the implementation step-by-step:

  • How to manage multi-threaded voice queues so your UI and camera feed don’t stutter while listening for audio.

  • Implementing state-machine logic to toggle cleanly between find, scan, and track modes.

  • Tuning servo gains, deadbands, and sweep step-sizes to achieve smooth, organic robotic movement without jitter.

  • Integrating real-time bounding box feedback, frames-per-second (FPS) monitoring, and dynamic visual overlays using OpenCV.

Grab your Raspberry Pi 5, fire up your Fusion Hat, and let’s write some code that brings hardware to life! Watch the full video lesson above, and let’s dive straight in!

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 58: Object Detection With Speech to Text Voice Control on Raspberry Pi 5

Welcome back, my friends! This is Paul McWhorter from toptechboy.com, and welcome to AI on the Edge LESSON 58: Object Detection With Speech to Text Voice Control on Raspberry Pi 5!

In this lesson, we take a massive leap forward in edge artificial intelligence and robotics by giving our Raspberry Pi 5 camera rig the gift of speech. Up until now, changing our target object of interest required editing our Python script and restarting the code. In Lesson 58, we bridge the gap between computer vision and natural human interaction by introducing local Speech-to-Text (STT) and Piper Text-to-Speech (TTS). Now, you can simply speak to your system, tell it what object you want to track, and watch as your pan-tilt camera locks on and follows your command!

Understanding the Program Architecture

The program is structured around a robust, multi-threaded architecture designed to handle audio recognition and high-speed computer vision simultaneously without lagging your video stream.

1. Voice-Driven Input via Background Threading

To ensure our camera stream maintains a smooth frame rate, we initialize a dedicated background worker thread using Python’s threading library. This thread runs independently, listening to microphone input via local STT tools.

  • When you speak your object of interest, the thread captures the audio, strips and lowercases the text, and checks for termination commands like quitting the program.

  • If a valid target is spoken, the thread pushes the string into a thread-safe communication queue and commands Piper TTS to talk back, audibly confirming your choice with a phrase like “Now Tracking”.

2. Thread-Safe Command Queuing

Because audio processing and computer vision operate on different timing loops, we use Python’s thread-safe queue mechanism. The background listening thread drops new target requests into the queue, while our main loop checks on every frame pass whether a new object of interest has arrived. This prevents race conditions and keeps our variable updates clean and reliable.

3. High-Speed OpenCV and MediaPipe Vision Loop

The main execution thread runs our primary while loop, capturing high-resolution frames from the Picamera2 camera module at 60 FPS, flipping the orientation, and feeding the frame data into MediaPipe’s efficient object detection model.

  • When detections match our active target class, the program extracts bounding box coordinates, calculates the geometric center of the object, renders visual confirmation boxes and confidence percentages on the screen, and computes pixel error offsets from the center of the display.

4. Proportional Servo Feedback Control

Once the pixel error between the target center and the screen center is calculated, our servo update function kicks in. By applying proportional gain multipliers to the horizontal and vertical error values, the script smoothly drives the SunFounder Fusion HAT servos, keeping your target locked right in the crosshairs.

Hardware Requirements

  • Raspberry Pi 5

  • Raspberry Pi Camera Module (configured with Picamera2 at 1280 by 720 resolution and 60 FPS)

  • SunFounder Fusion HAT (for driving pan and tilt servos)

  • USB Microphone (for local speech-to-text input)

  • Speaker or Headphones (for Piper text-to-speech audio feedback)

Homework Assignment for Next Week

Now that you have full voice-controlled object selection up and running, it is time for your next big challenge. For next week’s lesson, take the code provided at the bottom of this page and add mode control with three distinct operating states:

  1. find mode: In this mode, the servos remain completely switched off. The camera stays stationary, but MediaPipe actively finds, labels, and boxes your object of interest whenever it appears in the frame.

  2. track mode: In this mode, the servos are switched on, and the camera actively drives the pan-tilt rig to track and center the object of interest.

  3. scan mode: In this mode, if your target is out of view, the system actively sweeps and scans the room using the servos to find the object, and then automatically switches over to track mode once the object is located.

Grab your coffee, fire up your Raspberry Pi 5, review the complete code at the bottom of this page, and I will see you in the next lesson!

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 57: Detecting and Tracking Objects with MediaPipe

Welcome back, my friends! This is Paul McWhorter from toptechboy.com, and welcome to LESSON 57 of our AI on the Edge tutorial series.

In this lesson, we take what we’ve learned about local object detection using MediaPipe and put it to work in the physical world! We are building an active target tracking system using a 2-DOF pan-tilt servo rig driven by the Raspberry Pi and the Fusion HAT.

By calculating the positional error between the center of our targeted object (in this case, a bottle) and the center of our frame, we compute feedback adjustments to continuously drive our pan and tilt servos—keeping the target locked right in the center of the camera view.

What You Will Learn in Lesson 57

  1. Setting Up MediaPipe Object Detection: Configuring the efficientdet_lite0.tflite model with threshold options and maximum result limits.

  2. Integrating Picamera2 with OpenCV: Capturing high-resolution camera feeds (1280 by 720 at 60 FPS) and converting raw frames to MediaPipe-compatible image formats.

  3. Bounding Box Analysis & Centroid Calculation: Extracting bounding box dimensions and computing the precise pixel center of detected targets.

  4. Proportional Error Servo Control: Using gain coefficients to convert pixel displacements into physical angle corrections for pan and tilt servos.

  5. Deadband Implementation: Preventing servo chatter and hunting by defining a deadband region around the center of the screen.

Hardware Requirements

  • Raspberry Pi 5 (or Raspberry Pi 4)

  • Raspberry Pi Camera Module (configured with Picamera2)

  • SunFounder Fusion HAT (or compatible servo controller HAT)

  • 2-DOF Pan-Tilt Servo Kit (Pan attached to Pin 2, Tilt attached to Pin 3)

  • Target Object (e.g., a water bottle or any object recognized by COCO dataset categories)

How the Proportional Tracking Control Works

To keep the target centered in the video frame, we continuously measure the distance in pixels between the frame’s center and the center of the target’s bounding box:

errorX = boxCenterX – screenCenterX

errorY = boxCenterY – screenCenterY

If the magnitude of the error exceeds our deadBand limit (15 pixels), we adjust the pan and tilt angles proportionally:

panAngle = panAngle – (errorX * panGain)

tiltAngle = tiltAngle + (errorY * tiltGain)

  • Pan Limits: Clamped between -90 and +90 degrees.

  • Tilt Limits: Clamped between -45 and +45 degrees.

Homework Assignment

Now that you have basic single-target tracking running smoothly:

  1. Multi-Target Priority: Modify the script so that if multiple target objects appear on screen, the camera locks onto the object closest to the center of the frame rather than defaulting to the first detected bounding box.

  2. Dynamic Smoothness Filter: Add an exponential moving average or low-pass filter to the bounding box center coordinates to smooth out tracking jitter when the object moves rapidly across the field of view.

  3. Audibly speak the Object of Interst to track. This will require you to incorporate Speech to Text voice recognition.

Get your coffee, fire up your Raspberry Pi, copy the code above into your IDE, and let’s get tracking!

General Artificial Intelligence & Edge Computing Concepts

What is Edge AI?

Edge AI refers to the deployment of artificial intelligence algorithms directly on hardware devices, such as microcontrollers, single-board computers, or edge accelerators, located close to the source of data generation rather than relying on cloud-based processing servers.

Key Advantages of Local Inference:

  • Low Latency: Processing video feeds directly on an edge device eliminates network transmission delay, enabling real-time feedback loops critical for robotics and physical pan-tilt tracking.

  • Privacy & Security: Sensitive visual or audio data remains on the local hardware without requiring continuous streaming to third-party cloud infrastructure.

  • Network Independence: Edge deployments operate reliably in remote or bandwidth-constrained environments without requiring active internet connectivity.

  • Cost Efficiency: Running localized models eliminates recurring per-API-call cloud infrastructure and compute costs.

Proportional-Integral-Derivative Control Concepts:

  • Proportional Control: Calculates an output value proportional to the current error magnitude. High proportional gain results in aggressive responses, while low gain yields sluggish tracking.

  • Deadband: A non-responsive range around the zero-error position where control adjustments are intentionally suspended to prevent motor oscillation, noise amplification, and mechanical wear.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 56: Identifying Object of Interest in Target Zone with Mediapipe

Welcome back, everyone! Paul McWhorter here from toptechboy.com, and welcome to LESSON 56 of our AI on the Edge tutorial series! Grab yourself a nice hot cup of iced coffee (no sugar needed!), sit back, relax, and let’s jump right into today’s project.

AI on the Edge LESSON 56: Identifying Object of Interest in Target Zone with MediaPipe

In our previous lessons, we learned how to use MediaPipe’s object detection task to scan our live camera feed and pick out objects using pre-trained edge AI models like efficientdet_lite0.tflite. We got our bounding boxes drawn, displayed class labels, and showed confidence scores—all running smoothly right on our Raspberry Pi!

Today, we are taking our computer vision setup to the next level by building a Target Zone Detector.

What We Are Doing Today

Instead of just blindly detecting everything in the frame, we want our edge device to actively monitor a specific region of interest—a “Target Zone”—and trigger an alert when a specific Target Class enters that boundary.

Here is what we accomplish in this lesson:

  1. Define a Fixed Target Zone: We calculate a customizable bounding box right in the center of our 1280×720 video feed (e.g., 150×250 pixels).

  2. Filter by Object Class: We set a target object (in this example, a ‘bottle’). The system ignores all other object classes detected in the scene.

  3. Calculate the Bounding Box Center: For the target object, we compute its exact center point (X_center, Y_center) on the fly and draw a tracking dot on it.

  4. Boundary & Collision Logic: We test whether the center coordinates of our target fall inside our designated target zone using the condition: zoneX1 <= X_center <= zoneX2 and zoneY1 <= Y_center <= zoneY2.

  5. Dynamic Visual Feedback:

    • Idle State: When no target is inside the zone, the box remains Green.

    • Active State: The moment the target’s center point steps into the target zone, the boundary box turns Red, and an alert banner—”Target Has Entered the Zone”—is dynamically rendered on the camera feed.

  6. Real-Time Performance: We track frame rate (FPS) using an exponential moving average calculation to keep an eye on edge processing speed.

Homework / Challenge Assignment

Now that you have seen how to set up logic for a single target object in a single zone, here is your homework assignment to solidify what you’ve learned!

  1. We want to now actively track the selected Object of Interest. Turn your servos on on the pan tilt camera, and keep the Object of Interest in the center of the camera view.

Give it a try on your own before watching the solution in the next video. Remember: You don’t learn by watching; you learn by doing!

Discussion & Community

Did your target zone detection work smoothly on your Raspberry Pi setup? How many FPS are you getting with your Pi Camera? Leave a comment below on YouTube to share your results, questions, or modifications!

If you enjoyed this tutorial, hit that Like button, Subscribe to the channel, and share it with anyone who wants to learn real-world AI on the Edge.

As always, I’ll see you guys in the next lesson!

General Knowledge / Context

  • MediaPipe Tasks API: MediaPipe Python Tasks API provides high-level Python interfaces for object detection, landmasking, and classification tasks utilizing lightweight edge models such as TensorFlow Lite (.tflite).

  • PiCamera2 & OpenCV: In modern Raspberry Pi OS (Bullseye/Bookworm), Picamera2 handles native camera hardware capture in raw formats like RGB888, which can then be converted to MediaPipe Image objects (mp.Image) and rendered on OpenCV (cv2) GUI windows seamlessly.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 55: Object Detection on the Raspberry Pi 5 With MediaPipe

Welcome back, everyone! This is Paul McWhorter from TopTechBoy.com, and welcome to Lesson 55 in our AI on the Edge tutorial series!

If you’ve been following along, grab yourself a nice, hot cup of black coffee, fire up your Raspberry Pi 5, and let’s get right into it!

What We Are Doing in Lesson 55

Up to this point in the series, we have spent a lot of time working with land-marking models—tracking human hands, pose points, and facial features. But today, we step into full-blown computer vision object detection!

In this lesson, we learn how to implement MediaPipe’s lightweight, standalone object detection task API (mediapipe.tasks.python.vision.ObjectDetector) running on the Raspberry Pi 5. Using the pre-trained COCO dataset model (efficientdet_lite0.tflite), our Pi 5 can instantly identify up to 80 different real-world objects—from people, cups, and bottles to chairs, laptops, and cell phones—and draw real-time bounding boxes around them on our video feed.

Key Takeaways & Concepts Covered

  1. Setting Up the MediaPipe Object Detector Task API:

    Unlike older mp.solutions wrappers, the modern MediaPipe Tasks API requires configuring BaseOptions with the explicit local path to your .tflite model asset (modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite").

  2. Configuring Detector Parameters:

    • score_threshold (0.35): Sets the minimum confidence score required to accept a detected object. Lowering this value captures weaker detections but risks false positives; raising it ensures high-confidence target lock.

    • max_results (3): Limits the maximum number of bounding boxes predicted per frame to keep our processing loop lean and fast.

  3. Decoding Detection Bounding Boxes:

    MediaPipe returns detected objects with bounding box geometry relative to the frame origin (origin_x, origin_y, width, height). We parse these dimensions directly into OpenCV integers (x, y, w, h) to draw bounding boxes with cv2.rectangle().

  4. Category Labeling & Confidence Rendering:

    We pull category names (category_name) and classification scores (category.score) from detection.categories[0], format the confidence as a percentage, and render clean text labels anchored directly above each target box.

  5. Optimal Frame Alignment & FPS Monitoring:

    We capture 60 FPS video directly from picamera2 in RGB888 format, flip it properly to match our physical camera mounting, and track real-time frames-per-second using our low-pass smoothing formula.

What You Will Need

To follow along with this lesson, ensure you have your standard gear ready from the kit:

  • Raspberry Pi 5 (with Active Cooler/Heat Sink)

  • Raspberry Pi Camera Module 3 (or compatible Camera Module driven via picamera2)

  • SunFounder Fusion AI Lab Kit (with pre-downloaded MediaPipe .tflite model weights)

Homework Assignment

Alright, class, you know I never let you off the hook without some homework!

Your assignment for this lesson is to take this foundational object detection script and upgrade its intelligence:

  1. Target Filter: Modify the loop so that it only draws bounding boxes around a specific target object of your choice (for instance, only detect "person" or only detect "bottle"), ignoring all other detected COCO classes.

  2. Target Tracking Dot: Calculate the exact center pixel coordinate $(x + \frac{w}{2}, y + \frac{h}{2})$ of your targeted object, and draw a solid red target circle (cv2.circle) right on the object’s center point.

  3. Dynamic On-Screen Target Status: If your target object is visible in the frame, display a green text banner in the upper corner reading "TARGET ACQUIRED". If it leaves the frame, change the status banner to a red "SEARCHING...".

Give it a shot, take your time, and write your solution from scratch without copying and pasting! Drop a comment on the YouTube video to let me know if you got it working or if you ran into any snags.

Until next time, get your coffee, keep coding, and I’ll see you guys in Lesson 56!

 

Making The World a Better Place One High Tech Project at a Time. Enjoy!