Welcome back, my friends! This is Paul McWhorter from toptechboy.com, and welcome to AI on the Edge LESSON 58: Object Detection With Speech to Text Voice Control on Raspberry Pi 5!

In this lesson, we take a massive leap forward in edge artificial intelligence and robotics by giving our Raspberry Pi 5 camera rig the gift of speech. Up until now, changing our target object of interest required editing our Python script and restarting the code. In Lesson 58, we bridge the gap between computer vision and natural human interaction by introducing local Speech-to-Text (STT) and Piper Text-to-Speech (TTS). Now, you can simply speak to your system, tell it what object you want to track, and watch as your pan-tilt camera locks on and follows your command!

Understanding the Program Architecture

The program is structured around a robust, multi-threaded architecture designed to handle audio recognition and high-speed computer vision simultaneously without lagging your video stream.

1. Voice-Driven Input via Background Threading

To ensure our camera stream maintains a smooth frame rate, we initialize a dedicated background worker thread using Python’s threading library. This thread runs independently, listening to microphone input via local STT tools.

When you speak your object of interest, the thread captures the audio, strips and lowercases the text, and checks for termination commands like quitting the program.
If a valid target is spoken, the thread pushes the string into a thread-safe communication queue and commands Piper TTS to talk back, audibly confirming your choice with a phrase like “Now Tracking”.

2. Thread-Safe Command Queuing

Because audio processing and computer vision operate on different timing loops, we use Python’s thread-safe queue mechanism. The background listening thread drops new target requests into the queue, while our main loop checks on every frame pass whether a new object of interest has arrived. This prevents race conditions and keeps our variable updates clean and reliable.

3. High-Speed OpenCV and MediaPipe Vision Loop

The main execution thread runs our primary while loop, capturing high-resolution frames from the Picamera2 camera module at 60 FPS, flipping the orientation, and feeding the frame data into MediaPipe’s efficient object detection model.

When detections match our active target class, the program extracts bounding box coordinates, calculates the geometric center of the object, renders visual confirmation boxes and confidence percentages on the screen, and computes pixel error offsets from the center of the display.

4. Proportional Servo Feedback Control

Once the pixel error between the target center and the screen center is calculated, our servo update function kicks in. By applying proportional gain multipliers to the horizontal and vertical error values, the script smoothly drives the SunFounder Fusion HAT servos, keeping your target locked right in the crosshairs.

Hardware Requirements

Raspberry Pi 5
Raspberry Pi Camera Module (configured with Picamera2 at 1280 by 720 resolution and 60 FPS)
SunFounder Fusion HAT (for driving pan and tilt servos)
USB Microphone (for local speech-to-text input)
Speaker or Headphones (for Piper text-to-speech audio feedback)

Homework Assignment for Next Week

Now that you have full voice-controlled object selection up and running, it is time for your next big challenge. For next week’s lesson, take the code provided at the bottom of this page and add mode control with three distinct operating states:

find mode: In this mode, the servos remain completely switched off. The camera stays stationary, but MediaPipe actively finds, labels, and boxes your object of interest whenever it appears in the frame.
track mode: In this mode, the servos are switched on, and the camera actively drives the pan-tilt rig to track and center the object of interest.
scan mode: In this mode, if your target is out of view, the system actively sweeps and scans the room using the servos to find the object, and then automatically switches over to track mode once the object is located.

Grab your coffee, fire up your Raspberry Pi 5, review the complete code at the bottom of this page, and I will see you in the next lesson!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
from fusion_hat.servo import Servo

from fusion_hat.stt import STT
from fusion_hat.tts import Piper
import threading
from queue import Queue

tts = Piper()
tts.set_model('en_US-kristin-medium')

stt = STT('en-us')
ooiQueue = Queue()
running = True


piCam = Picamera2(1)
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)
topBar = 65
windowPadding = 0
cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)
cv2.resizeWindow('Camera', W, H)
tSettle = .1
time.sleep(tSettle)
cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"
scoreThreshold = .35
maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)
options = vision.ObjectDetectorOptions(base_options = baseOptions,
                                       score_threshold=scoreThreshold,
                                       max_results=maxObj)
myDetector = vision.ObjectDetector.create_from_options(options)

targetClass = None

panPin = 2
tiltPin = 3
panServo = Servo(panPin)
tiltServo = Servo(tiltPin)
panAngle = 0
tiltAngle = 0
panServo.angle(panAngle)
tiltServo.angle(0)

panGain = .015
tiltGain = .015
deadBand = 15

def listenForOoiThread():
    global running
    print("Voice Listener is Running")
    while running:
        spokenText = stt.listen(stream=False)
        if spokenText:
            cleanText = spokenText.strip().lower()
            print("Heard OOI Request: ", cleanText)
            if cleanText == 'quit':
                running = False
                exitMsg = "Terminating Objest Tracking Program, Have a Nice Day"
                tts.say(exitMsg, stream = False)
            ooiQueue.put(cleanText)
            confirmMsg = "Now Tracking: "+cleanText
            tts.say(confirmMsg,stream=False)
    print("Voice Litener Thread Terminated")
                
voiceThread = threading.Thread(target=listenForOoiThread, daemon=False)
voiceThread.start()

def updateServo(objectCenterX,objectCenterY, width, height):
    global panAngle, tiltAngle
    tSettle = .01
    screenCenterX = width/2
    screenCenterY = height/2
    
    errorX = objectCenterX - screenCenterX
    errorY = objectCenterY - screenCenterY
    
    if abs(errorX) > deadBand:
        panAngle = panAngle - (errorX*panGain)
        if panAngle > 90:
            panAngle = 90
        if panAngle < -90:
            panAngle = -90
        panServo.angle(panAngle)
    if abs(errorY) > deadBand:
        tiltAngle = tiltAngle + (errorY*tiltGain)
        if tiltAngle > 45:
            tiltAngle = 45
        if tiltAngle < -45:
            tiltAngle = -45
        tiltServo.angle(tiltAngle)
        time.sleep(tSettle)
tWait = 1
time.sleep(tWait)
welcomeMsg = "Please speak your object of interest to track"  
tts.say(welcomeMsg,stream = False)

while running:
    if not ooiQueue.empty():
        targetClass = ooiQueue.get()
        print("Active OOI Updated to: ",targetClass)
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)
    myDetections = myDetector.detect(myImage)
    targetInZone = False
    #print(myDetections.detections)
    cv2.circle(frame,(int(W/2),int(H/2)),deadBand*2,(0,255,0),2)
    for detection in myDetections.detections:
        if detection.categories:
            category=detection.categories[0]
            if category.category_name == targetClass:
                box = detection.bounding_box
                x = int(box.origin_x)
                y = int(box.origin_y)
                w = int(box.width)
                h = int(box.height)
                boxCenterX = x + int(w/2)
                boxCenterY = y + int(h/2)
                cv2.rectangle(frame,(x,y),(x+w,y+h),
                              (255,0,0),fontThickness)
                labelText = "Target Aquired "+str(round(category.score*100))+'%'
                cv2.circle(frame,(boxCenterX,boxCenterY),6,(0,0,255),-1)
                cv2.putText(frame,labelText,(x,y-10),fontFace,fontScale*.8,(0,255,0),2)
                updateServo(boxCenterX,boxCenterY,W,H)
                                     
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    #cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
myDetector.close()
cv2.destroyAllWindows()
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

from mediapipe.tasks import python

from mediapipe.tasks.python import vision

from fusion_hat.servo import Servo

from fusion_hat.stt import STT

from fusion_hat.tts import Piper

import threading

from queue import Queue

tts = Piper()

tts.set_model('en_US-kristin-medium')

stt = STT('en-us')

ooiQueue = Queue()

running = True

piCam = Picamera2(1)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

topBar = 65

windowPadding = 0

cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)

cv2.resizeWindow('Camera', W, H)

tSettle = .1

time.sleep(tSettle)

cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"

scoreThreshold = .35

maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)

options = vision.ObjectDetectorOptions(base_options = baseOptions,

score_threshold=scoreThreshold,

max_results=maxObj)

myDetector = vision.ObjectDetector.create_from_options(options)

targetClass = None

panPin = 2

tiltPin = 3

panServo = Servo(panPin)

tiltServo = Servo(tiltPin)

panAngle = 0

tiltAngle = 0

panServo.angle(panAngle)

tiltServo.angle(0)

panGain = .015

tiltGain = .015

deadBand = 15

def listenForOoiThread():

global running

print("Voice Listener is Running")

while running:

spokenText = stt.listen(stream=False)

if spokenText:

cleanText = spokenText.strip().lower()

print("Heard OOI Request: ", cleanText)

if cleanText == 'quit':

running = False

exitMsg = "Terminating Objest Tracking Program, Have a Nice Day"

tts.say(exitMsg, stream = False)

ooiQueue.put(cleanText)

confirmMsg = "Now Tracking: "+cleanText

tts.say(confirmMsg,stream=False)

print("Voice Litener Thread Terminated")

voiceThread = threading.Thread(target=listenForOoiThread, daemon=False)

voiceThread.start()

def updateServo(objectCenterX,objectCenterY, width, height):

global panAngle, tiltAngle

tSettle = .01

screenCenterX = width/2

screenCenterY = height/2

errorX = objectCenterX - screenCenterX

errorY = objectCenterY - screenCenterY

if abs(errorX) > deadBand:

panAngle = panAngle - (errorX*panGain)

if panAngle > 90:

panAngle = 90

if panAngle < -90:

panAngle = -90

panServo.angle(panAngle)

if abs(errorY) > deadBand:

tiltAngle = tiltAngle + (errorY*tiltGain)

if tiltAngle > 45:

tiltAngle = 45

if tiltAngle < -45:

tiltAngle = -45

tiltServo.angle(tiltAngle)

time.sleep(tSettle)

tWait = 1

time.sleep(tWait)

welcomeMsg = "Please speak your object of interest to track"

tts.say(welcomeMsg,stream = False)

while running:

if not ooiQueue.empty():

targetClass = ooiQueue.get()

print("Active OOI Updated to: ",targetClass)

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)

myDetections = myDetector.detect(myImage)

targetInZone = False

#print(myDetections.detections)

cv2.circle(frame,(int(W/2),int(H/2)),deadBand*2,(0,255,0),2)

for detection in myDetections.detections:

if detection.categories:

category=detection.categories[0]

if category.category_name == targetClass:

box = detection.bounding_box

x = int(box.origin_x)

y = int(box.origin_y)

w = int(box.width)

h = int(box.height)

boxCenterX = x + int(w/2)

boxCenterY = y + int(h/2)

cv2.rectangle(frame,(x,y),(x+w,y+h),

(255,0,0),fontThickness)

labelText = "Target Aquired "+str(round(category.score*100))+'%'

cv2.circle(frame,(boxCenterX,boxCenterY),6,(0,0,255),-1)

cv2.putText(frame,labelText,(x,y-10),fontFace,fontScale*.8,(0,255,0),2)

updateServo(boxCenterX,boxCenterY,W,H)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

#cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

myDetector.close()

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 57: Detecting and Tracking Objects with MediaPipe

November 1, 2026 admin

Welcome back, my friends! This is Paul McWhorter from toptechboy.com, and welcome to LESSON 57 of our AI on the Edge tutorial series.

In this lesson, we take what we’ve learned about local object detection using MediaPipe and put it to work in the physical world! We are building an active target tracking system using a 2-DOF pan-tilt servo rig driven by the Raspberry Pi and the Fusion HAT.

By calculating the positional error between the center of our targeted object (in this case, a bottle) and the center of our frame, we compute feedback adjustments to continuously drive our pan and tilt servos—keeping the target locked right in the center of the camera view.

What You Will Learn in Lesson 57

Setting Up MediaPipe Object Detection: Configuring the efficientdet_lite0.tflite model with threshold options and maximum result limits.
Integrating Picamera2 with OpenCV: Capturing high-resolution camera feeds (1280 by 720 at 60 FPS) and converting raw frames to MediaPipe-compatible image formats.
Bounding Box Analysis & Centroid Calculation: Extracting bounding box dimensions and computing the precise pixel center of detected targets.
Proportional Error Servo Control: Using gain coefficients to convert pixel displacements into physical angle corrections for pan and tilt servos.
Deadband Implementation: Preventing servo chatter and hunting by defining a deadband region around the center of the screen.

Hardware Requirements

Raspberry Pi 5 (or Raspberry Pi 4)
Raspberry Pi Camera Module (configured with Picamera2)
SunFounder Fusion HAT (or compatible servo controller HAT)
2-DOF Pan-Tilt Servo Kit (Pan attached to Pin 2, Tilt attached to Pin 3)
Target Object (e.g., a water bottle or any object recognized by COCO dataset categories)

How the Proportional Tracking Control Works

To keep the target centered in the video frame, we continuously measure the distance in pixels between the frame’s center and the center of the target’s bounding box:

errorX = boxCenterX – screenCenterX

errorY = boxCenterY – screenCenterY

If the magnitude of the error exceeds our deadBand limit (15 pixels), we adjust the pan and tilt angles proportionally:

panAngle = panAngle – (errorX * panGain)

tiltAngle = tiltAngle + (errorY * tiltGain)

Pan Limits: Clamped between -90 and +90 degrees.
Tilt Limits: Clamped between -45 and +45 degrees.

Homework Assignment

Now that you have basic single-target tracking running smoothly:

Multi-Target Priority: Modify the script so that if multiple target objects appear on screen, the camera locks onto the object closest to the center of the frame rather than defaulting to the first detected bounding box.
Dynamic Smoothness Filter: Add an exponential moving average or low-pass filter to the bounding box center coordinates to smooth out tracking jitter when the object moves rapidly across the field of view.
Audibly speak the Object of Interst to track. This will require you to incorporate Speech to Text voice recognition.

Get your coffee, fire up your Raspberry Pi, copy the code above into your IDE, and let’s get tracking!

General Artificial Intelligence & Edge Computing Concepts

What is Edge AI?

Edge AI refers to the deployment of artificial intelligence algorithms directly on hardware devices, such as microcontrollers, single-board computers, or edge accelerators, located close to the source of data generation rather than relying on cloud-based processing servers.

Key Advantages of Local Inference:

Low Latency: Processing video feeds directly on an edge device eliminates network transmission delay, enabling real-time feedback loops critical for robotics and physical pan-tilt tracking.
Privacy & Security: Sensitive visual or audio data remains on the local hardware without requiring continuous streaming to third-party cloud infrastructure.
Network Independence: Edge deployments operate reliably in remote or bandwidth-constrained environments without requiring active internet connectivity.
Cost Efficiency: Running localized models eliminates recurring per-API-call cloud infrastructure and compute costs.

Proportional-Integral-Derivative Control Concepts:

Proportional Control: Calculates an output value proportional to the current error magnitude. High proportional gain results in aggressive responses, while low gain yields sluggish tracking.
Deadband: A non-responsive range around the zero-error position where control adjustments are intentionally suspended to prevent motor oscillation, noise amplification, and mechanical wear.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
from fusion_hat.servo import Servo
piCam = Picamera2(1)
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)
topBar = 65
windowPadding = 0
cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)
cv2.resizeWindow('Camera', W, H)
tSettle = .1
time.sleep(tSettle)
cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"
scoreThreshold = .35
maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)
options = vision.ObjectDetectorOptions(base_options = baseOptions,
                                       score_threshold=scoreThreshold,
                                       max_results=maxObj)
myDetector = vision.ObjectDetector.create_from_options(options)

targetClass = 'bottle'

panPin = 2
tiltPin = 3
panServo = Servo(panPin)
tiltServo = Servo(tiltPin)
panAngle = 0
tiltAngle = 0
panServo.angle(panAngle)
tiltServo.angle(0)

panGain = .015
tiltGain = .015
deadBand = 15

def updateServo(objectCenterX,objectCenterY, width, height):
    global panAngle, tiltAngle
    tSettle = .01
    screenCenterX = width/2
    screenCenterY = height/2
    
    errorX = objectCenterX - screenCenterX
    errorY = objectCenterY - screenCenterY
    
    if abs(errorX) > deadBand:
        panAngle = panAngle - (errorX*panGain)
        if panAngle > 90:
            panAngle = 90
        if panAngle < -90:
            panAngle = -90
        panServo.angle(panAngle)
    if abs(errorY) > deadBand:
        tiltAngle = tiltAngle + (errorY*tiltGain)
        if tiltAngle > 45:
            tiltAngle = 45
        if tiltAngle < -45:
            tiltAngle = -45
        tiltServo.angle(tiltAngle)
        time.sleep(tSettle)
        
            
while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)
    myDetections = myDetector.detect(myImage)
    targetInZone = False
    #print(myDetections.detections)
    cv2.circle(frame,(int(W/2),int(H/2)),deadBand*2,(0,255,0),2)
    for detection in myDetections.detections:
        if detection.categories:
            category=detection.categories[0]
            if category.category_name == targetClass:
                box = detection.bounding_box
                x = int(box.origin_x)
                y = int(box.origin_y)
                w = int(box.width)
                h = int(box.height)
                boxCenterX = x + int(w/2)
                boxCenterY = y + int(h/2)
                cv2.rectangle(frame,(x,y),(x+w,y+h),
                              (255,0,0),fontThickness)
                labelText = "Target Aquired "+str(round(category.score*100))+'%'
                cv2.circle(frame,(boxCenterX,boxCenterY),6,(0,0,255),-1)
                cv2.putText(frame,labelText,(x,y-10),fontFace,fontScale*.8,(0,255,0),2)
                updateServo(boxCenterX,boxCenterY,W,H)
                                     
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    #cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
myDetector.close()
cv2.destroyAllWindows()
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

from mediapipe.tasks import python

from mediapipe.tasks.python import vision

from fusion_hat.servo import Servo

piCam = Picamera2(1)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

topBar = 65

windowPadding = 0

cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)

cv2.resizeWindow('Camera', W, H)

tSettle = .1

time.sleep(tSettle)

cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"

scoreThreshold = .35

maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)

options = vision.ObjectDetectorOptions(base_options = baseOptions,

score_threshold=scoreThreshold,

max_results=maxObj)

myDetector = vision.ObjectDetector.create_from_options(options)

targetClass = 'bottle'

panPin = 2

tiltPin = 3

panServo = Servo(panPin)

tiltServo = Servo(tiltPin)

panAngle = 0

tiltAngle = 0

panServo.angle(panAngle)

tiltServo.angle(0)

panGain = .015

tiltGain = .015

deadBand = 15

def updateServo(objectCenterX,objectCenterY, width, height):

global panAngle, tiltAngle

tSettle = .01

screenCenterX = width/2

screenCenterY = height/2

errorX = objectCenterX - screenCenterX

errorY = objectCenterY - screenCenterY

if abs(errorX) > deadBand:

panAngle = panAngle - (errorX*panGain)

if panAngle > 90:

panAngle = 90

if panAngle < -90:

panAngle = -90

panServo.angle(panAngle)

if abs(errorY) > deadBand:

tiltAngle = tiltAngle + (errorY*tiltGain)

if tiltAngle > 45:

tiltAngle = 45

if tiltAngle < -45:

tiltAngle = -45

tiltServo.angle(tiltAngle)

time.sleep(tSettle)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)

myDetections = myDetector.detect(myImage)

targetInZone = False

#print(myDetections.detections)

cv2.circle(frame,(int(W/2),int(H/2)),deadBand*2,(0,255,0),2)

for detection in myDetections.detections:

if detection.categories:

category=detection.categories[0]

if category.category_name == targetClass:

box = detection.bounding_box

x = int(box.origin_x)

y = int(box.origin_y)

w = int(box.width)

h = int(box.height)

boxCenterX = x + int(w/2)

boxCenterY = y + int(h/2)

cv2.rectangle(frame,(x,y),(x+w,y+h),

(255,0,0),fontThickness)

labelText = "Target Aquired "+str(round(category.score*100))+'%'

cv2.circle(frame,(boxCenterX,boxCenterY),6,(0,0,255),-1)

cv2.putText(frame,labelText,(x,y-10),fontFace,fontScale*.8,(0,255,0),2)

updateServo(boxCenterX,boxCenterY,W,H)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

#cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

myDetector.close()

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 56: Identifying Object of Interest in Target Zone with Mediapipe

October 26, 2026 admin

Welcome back, everyone! Paul McWhorter here from toptechboy.com, and welcome to LESSON 56 of our AI on the Edge tutorial series! Grab yourself a nice hot cup of iced coffee (no sugar needed!), sit back, relax, and let’s jump right into today’s project.

AI on the Edge LESSON 56: Identifying Object of Interest in Target Zone with MediaPipe

In our previous lessons, we learned how to use MediaPipe’s object detection task to scan our live camera feed and pick out objects using pre-trained edge AI models like efficientdet_lite0.tflite. We got our bounding boxes drawn, displayed class labels, and showed confidence scores—all running smoothly right on our Raspberry Pi!

Today, we are taking our computer vision setup to the next level by building a Target Zone Detector.

What We Are Doing Today

Instead of just blindly detecting everything in the frame, we want our edge device to actively monitor a specific region of interest—a “Target Zone”—and trigger an alert when a specific Target Class enters that boundary.

Here is what we accomplish in this lesson:

Define a Fixed Target Zone: We calculate a customizable bounding box right in the center of our 1280×720 video feed (e.g., 150×250 pixels).
Filter by Object Class: We set a target object (in this example, a ‘bottle’). The system ignores all other object classes detected in the scene.
Calculate the Bounding Box Center: For the target object, we compute its exact center point (X_center, Y_center) on the fly and draw a tracking dot on it.
Boundary & Collision Logic: We test whether the center coordinates of our target fall inside our designated target zone using the condition: zoneX1 <= X_center <= zoneX2 and zoneY1 <= Y_center <= zoneY2.
Dynamic Visual Feedback:
- Idle State: When no target is inside the zone, the box remains Green.
- Active State: The moment the target’s center point steps into the target zone, the boundary box turns Red, and an alert banner—”Target Has Entered the Zone”—is dynamically rendered on the camera feed.
Real-Time Performance: We track frame rate (FPS) using an exponential moving average calculation to keep an eye on edge processing speed.

Homework / Challenge Assignment

Now that you have seen how to set up logic for a single target object in a single zone, here is your homework assignment to solidify what you’ve learned!

We want to now actively track the selected Object of Interest. Turn your servos on on the pan tilt camera, and keep the Object of Interest in the center of the camera view.

Give it a try on your own before watching the solution in the next video. Remember: You don’t learn by watching; you learn by doing!

Discussion & Community

Did your target zone detection work smoothly on your Raspberry Pi setup? How many FPS are you getting with your Pi Camera? Leave a comment below on YouTube to share your results, questions, or modifications!

If you enjoyed this tutorial, hit that Like button, Subscribe to the channel, and share it with anyone who wants to learn real-world AI on the Edge.

As always, I’ll see you guys in the next lesson!

General Knowledge / Context

MediaPipe Tasks API: MediaPipe Python Tasks API provides high-level Python interfaces for object detection, landmasking, and classification tasks utilizing lightweight edge models such as TensorFlow Lite (.tflite).
PiCamera2 & OpenCV: In modern Raspberry Pi OS (Bullseye/Bookworm), Picamera2 handles native camera hardware capture in raw formats like RGB888, which can then be converted to MediaPipe Image objects (mp.Image) and rendered on OpenCV (cv2) GUI windows seamlessly.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision

piCam = Picamera2(1)
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)
topBar = 65
windowPadding = 0
cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)
cv2.resizeWindow('Camera', W, H)
tSettle = .1
time.sleep(tSettle)
cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"
scoreThreshold = .35
maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)
options = vision.ObjectDetectorOptions(base_options = baseOptions,
                                       score_threshold=scoreThreshold,
                                       max_results=maxObj)
myDetector = vision.ObjectDetector.create_from_options(options)

targetClass = 'bottle'
zoneW = 150
zoneH = 250
zoneX1 = int(W/2 - zoneW/2)
zoneY1 = int(H/2 - zoneH/2)
zoneX2 = zoneX1 + zoneW
zoneY2 = zoneY1 + zoneH

zoneColorIdle = (0,255,0)
zoneColorActive = (0,0,255)


while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)
    myDetections = myDetector.detect(myImage)
    targetInZone = False
    #print(myDetections.detections)
    for detection in myDetections.detections:
        if detection.categories:
            category=detection.categories[0]
            if category.category_name == targetClass:
                box = detection.bounding_box
                x = int(box.origin_x)
                y = int(box.origin_y)
                w = int(box.width)
                h = int(box.height)
                boxCenterX = x + int(w/2)
                boxCenterY = y + int(h/2)
                cv2.rectangle(frame,(x,y),(x+w,y+h),
                              (255,0,0),fontThickness)
                labelText = "Target Aquired "+str(round(category.score*100))+'%'
                cv2.circle(frame,(boxCenterX,boxCenterY),6,(0,0,255),-1)
                cv2.putText(frame,labelText,(x,y-10),fontFace,fontScale*.8,(0,255,0),2)
                if (zoneX1<=boxCenterX<=zoneX2) and (zoneY1<=boxCenterY<=zoneY2):
                    targetInZone = True
            if targetInZone == True:
                cv2.rectangle(frame,(zoneX1,zoneY1),(zoneX2,zoneY2),zoneColorActive,fontThickness)
                bannerText = "Target Has Entered the Zone"
                cv2.putText(frame,bannerText,(zoneX1,zoneY1-25),
                            fontFace,fontScale*.9,zoneColorActive,2)
            if targetInZone == False:
                cv2.rectangle(frame,(zoneX1,zoneY1),(zoneX2,zoneY2),zoneColorIdle,fontThickness)
                
                                     
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    #cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
myDetector.close()
cv2.destroyAllWindows()
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

from mediapipe.tasks import python

from mediapipe.tasks.python import vision

piCam = Picamera2(1)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

topBar = 65

windowPadding = 0

cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)

cv2.resizeWindow('Camera', W, H)

tSettle = .1

time.sleep(tSettle)

cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"

scoreThreshold = .35

maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)

options = vision.ObjectDetectorOptions(base_options = baseOptions,

score_threshold=scoreThreshold,

max_results=maxObj)

myDetector = vision.ObjectDetector.create_from_options(options)

targetClass = 'bottle'

zoneW = 150

zoneH = 250

zoneX1 = int(W/2 - zoneW/2)

zoneY1 = int(H/2 - zoneH/2)

zoneX2 = zoneX1 + zoneW

zoneY2 = zoneY1 + zoneH

zoneColorIdle = (0,255,0)

zoneColorActive = (0,0,255)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)

myDetections = myDetector.detect(myImage)

targetInZone = False

#print(myDetections.detections)

for detection in myDetections.detections:

if detection.categories:

category=detection.categories[0]

if category.category_name == targetClass:

box = detection.bounding_box

x = int(box.origin_x)

y = int(box.origin_y)

w = int(box.width)

h = int(box.height)

boxCenterX = x + int(w/2)

boxCenterY = y + int(h/2)

cv2.rectangle(frame,(x,y),(x+w,y+h),

(255,0,0),fontThickness)

labelText = "Target Aquired "+str(round(category.score*100))+'%'

cv2.circle(frame,(boxCenterX,boxCenterY),6,(0,0,255),-1)

cv2.putText(frame,labelText,(x,y-10),fontFace,fontScale*.8,(0,255,0),2)

if (zoneX1<=boxCenterX<=zoneX2) and (zoneY1<=boxCenterY<=zoneY2):

targetInZone = True

if targetInZone == True:

cv2.rectangle(frame,(zoneX1,zoneY1),(zoneX2,zoneY2),zoneColorActive,fontThickness)

bannerText = "Target Has Entered the Zone"

cv2.putText(frame,bannerText,(zoneX1,zoneY1-25),

fontFace,fontScale*.9,zoneColorActive,2)

if targetInZone == False:

cv2.rectangle(frame,(zoneX1,zoneY1),(zoneX2,zoneY2),zoneColorIdle,fontThickness)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

#cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

myDetector.close()

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 55: Object Detection on the Raspberry Pi 5 With MediaPipe

October 17, 2026 admin

Welcome back, everyone! This is Paul McWhorter from TopTechBoy.com, and welcome to Lesson 55 in our AI on the Edge tutorial series!

If you’ve been following along, grab yourself a nice, hot cup of black coffee, fire up your Raspberry Pi 5, and let’s get right into it!

What We Are Doing in Lesson 55

Up to this point in the series, we have spent a lot of time working with land-marking models—tracking human hands, pose points, and facial features. But today, we step into full-blown computer vision object detection!

In this lesson, we learn how to implement MediaPipe’s lightweight, standalone object detection task API (mediapipe.tasks.python.vision.ObjectDetector) running on the Raspberry Pi 5. Using the pre-trained COCO dataset model (efficientdet_lite0.tflite), our Pi 5 can instantly identify up to 80 different real-world objects—from people, cups, and bottles to chairs, laptops, and cell phones—and draw real-time bounding boxes around them on our video feed.

Key Takeaways & Concepts Covered

Setting Up the MediaPipe Object Detector Task API:

Unlike older mp.solutions wrappers, the modern MediaPipe Tasks API requires configuring BaseOptions with the explicit local path to your .tflite model asset (modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite").
Configuring Detector Parameters:
- score_threshold (0.35): Sets the minimum confidence score required to accept a detected object. Lowering this value captures weaker detections but risks false positives; raising it ensures high-confidence target lock.
- max_results (3): Limits the maximum number of bounding boxes predicted per frame to keep our processing loop lean and fast.
Decoding Detection Bounding Boxes:

MediaPipe returns detected objects with bounding box geometry relative to the frame origin (origin_x, origin_y, width, height). We parse these dimensions directly into OpenCV integers (x, y, w, h) to draw bounding boxes with cv2.rectangle().
Category Labeling & Confidence Rendering:

We pull category names (category_name) and classification scores (category.score) from detection.categories[0], format the confidence as a percentage, and render clean text labels anchored directly above each target box.
Optimal Frame Alignment & FPS Monitoring:

We capture 60 FPS video directly from picamera2 in RGB888 format, flip it properly to match our physical camera mounting, and track real-time frames-per-second using our low-pass smoothing formula.

What You Will Need

To follow along with this lesson, ensure you have your standard gear ready from the kit:

Raspberry Pi 5 (with Active Cooler/Heat Sink)
Raspberry Pi Camera Module 3 (or compatible Camera Module driven via picamera2)
SunFounder Fusion AI Lab Kit (with pre-downloaded MediaPipe .tflite model weights)

Homework Assignment

Alright, class, you know I never let you off the hook without some homework!

Your assignment for this lesson is to take this foundational object detection script and upgrade its intelligence:

Target Filter: Modify the loop so that it only draws bounding boxes around a specific target object of your choice (for instance, only detect "person" or only detect "bottle"), ignoring all other detected COCO classes.
Target Tracking Dot: Calculate the exact center pixel coordinate $(x + \frac{w}{2}, y + \frac{h}{2})$ of your targeted object, and draw a solid red target circle (cv2.circle) right on the object’s center point.
Dynamic On-Screen Target Status: If your target object is visible in the frame, display a green text banner in the upper corner reading "TARGET ACQUIRED". If it leaves the frame, change the status banner to a red "SEARCHING...".

Give it a shot, take your time, and write your solution from scratch without copying and pasting! Drop a comment on the YouTube video to let me know if you got it working or if you ran into any snags.

Until next time, get your coffee, keep coding, and I’ll see you guys in Lesson 56!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision

piCam = Picamera2(1)
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)
topBar = 65
windowPadding = 0
cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)
cv2.resizeWindow('Camera', W, H)
tSettle = .1
time.sleep(tSettle)
cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"
scoreThreshold = .35
maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)
options = vision.ObjectDetectorOptions(base_options = baseOptions,
                                       score_threshold=scoreThreshold,
                                       max_results=maxObj)
myDetector = vision.ObjectDetector.create_from_options(options)

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)
    myDetections = myDetector.detect(myImage)
    #print(myDetections.detections)
    for detection in myDetections.detections:
        box = detection.bounding_box
        x = box.origin_x
        y = box.origin_y
        w = box.width
        h = box.height
        cv2.rectangle(frame, (x,y), (x+w,y+h), (255,0,0), fontThickness)
    if detection.categories:
        category = detection.categories[0]
        myLabel = category.category_name
        labelText = myLabel+" "+ str(round(category.score*100))+"%"
        cv2.putText(frame,labelText, (x, max(y-10,25)),fontFace,
                                      fontScale*.8,
                                      (0,255,0),
                                      2)
                                      
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    #cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
myDetector.close()
cv2.destroyAllWindows()
print('Program Terminated')

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

from mediapipe.tasks import python

from mediapipe.tasks.python import vision

piCam = Picamera2(1)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

topBar = 65

windowPadding = 0

cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)

cv2.resizeWindow('Camera', W, H)

tSettle = .1

time.sleep(tSettle)

cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"

scoreThreshold = .35

maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)

options = vision.ObjectDetectorOptions(base_options = baseOptions,

score_threshold=scoreThreshold,

max_results=maxObj)

myDetector = vision.ObjectDetector.create_from_options(options)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)

myDetections = myDetector.detect(myImage)

#print(myDetections.detections)

for detection in myDetections.detections:

box = detection.bounding_box

x = box.origin_x

y = box.origin_y

w = box.width

h = box.height

cv2.rectangle(frame, (x,y), (x+w,y+h), (255,0,0), fontThickness)

if detection.categories:

category = detection.categories[0]

myLabel = category.category_name

labelText = myLabel+" "+ str(round(category.score*100))+"%"

cv2.putText(frame,labelText, (x, max(y-10,25)),fontFace,

fontScale*.8,

(0,255,0),

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

#cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

myDetector.close()

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 54: Custom Hand Gesture Recognition & Dynamic NeoPixel Control

October 9, 2026 admin

Welcome back, everyone! In this lesson, we take edge AI interactivity to the next level. Up until now, we have learned how to track hand landmarks, extract pixel coordinates, and calculate distances using MediaPipe. Today, we put all those building blocks together into a self-contained, real-time hand gesture recognition system running on the Raspberry Pi 5.

We are not relying on a black-box pre-trained gesture model. Instead, we write our own mathematical feature extractor, train our custom hand poses on the fly, save those gesture profiles directly to disk, and map each gesture to a non-blocking lighting animation on a NeoPixel LED ring driven over SPI.

System Architecture Overview

The system operates in three distinct phases running simultaneously inside our main loop:

Normalized Feature Extraction: MediaPipe extracts 3D hand landmark coordinates. To make gesture detection scale-invariant and distance-invariant, we normalize the distances between the fingertips and the wrist against a reference palm dimension.
Nearest-Neighbor Gesture Matching: Live distance vectors are compared against stored gesture profiles using a mean absolute error algorithm. If the error falls below our maximum threshold, the gesture is identified; otherwise, it defaults to “Unknown.”
Non-Blocking Hardware Animation Dispatch: The recognized gesture string triggers corresponding pattern functions on the 12-LED NeoPixel ring via neopixel_spi. Animations use non-blocking time delta checks to preserve high camera preview frame rates.

Gesture-to-Hardware Animation Mapping

Our script contains predefined animation routines mapped to recognized gesture strings. Here is how each pose controls the LED ring:

Recognized Gesture	LED Animation Pattern	Visual Effect
`fist`	Red Pulse	Sinusoidal breathing effect fading in and out
`palm`	Solid Beacon	Full brightness white ring
`peace`	Dual Spinner	Dual green and blue dots rotating over a dim red background
`thumbsup`	Solid Success	Constant green feedback
`point`	Radar Sweep	A green trailing dot chasing around a red ring
`rock`	Cyan Flame	Sinusoidal green-cyan intensity modulation
`ok`	Rainbow Wheel	Rotating HSV spectrum cycle
`Unknown` / Unmatched	Clear	LEDs turn completely off

Interactive Keyboard Controls

While the camera loop is active, you can interact with the system using your keyboard in the preview window:

Spacebar: Captures the current hand pose’s normalized distance vector, assigns it to the active target gesture name, and saves the updated dataset to gestureNEOPIXEL.pkl.
t: Prompts the terminal to add a new custom gesture name to your training queue dynamically.
d: Prompts the terminal to delete a specific gesture profile from the saved dataset.
q: Clears the NeoPixel ring, closes the camera pipeline, saves training data, and terminates the program cleanly.

Hardware Wiring Reference

Ensure your Raspberry Pi 5 SPI interface is enabled before running the code (sudo raspi-config -> Interface Options -> SPI).

NeoPixel Data Input (DIN): Connect to GPIO 10 (SPI0 MOSI / Pin 19).
NeoPixel Ground: Connect to any GND pin on the Pi expansion header.
NeoPixel Power: Connect to an external 5V power supply (ensure a shared ground with the Raspberry Pi).

Homework Challenge

Now that you have a working gesture recognition system linked to hardware animations, try adding your own custom gestures! Train a brand-new gesture pose (like a “three-finger count”), write a corresponding non-blocking animation function in updateNeoPixels(), and share your video results.

Here is the code we developed in the video lesson:

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import math
import pickle
import os
import colorsys

# ====================================================================
# NEOPIXEL CONTROL INITIALIZATION (Adafruit neopixel_spi via SPI)
# ====================================================================

import board
import neopixel_spi as neopixel

NUM_LEDS = 12  # Set to match your NeoPixel ring size (e.g., 12)
PO = neopixel.GRB

# Initialize NeoPixel ring using SPI
spi = board.SPI()
strip = neopixel.NeoPixel_SPI(
    spi, 
    NUM_LEDS, 
    pixel_order=PO, 
    auto_write=False
)

# Turn off all LEDs on startup
strip.fill(0)
strip.show()

NEOPIXEL_ENABLED = True
print("[INFO] NeoPixel SPI initialized successfully.")

# Animation state variables (non-blocking updates)
animation_step = 0
lastAnimUpdate = time.time()
# ====================================================================

DATA_FILE = "gestureNEOPIXEL.pkl"

# Function to save the trained gesture dictionary to a file using pickle
def saveTrainedData(filename, data):
    with open(filename, 'wb') as f:
        pickle.dump(data, f)
    print("[INFO] Saved gesture data to " + str(filename))

# Function to load previously saved gesture data from a file
def loadTrainedData(filename):
    if os.path.exists(filename):
        with open(filename, 'rb') as f:
            data = pickle.load(f)
            print("[INFO] Loaded " + str(len(data)) + " gesture(s) from " + str(filename))
        return data
    return {}

# --- SETUP & INITIAL TRAINING ---
trainedData = loadTrainedData(DATA_FILE)
gestureNames = []

if not trainedData:
    print("HAND GESTURE TRAINING SETUP")
    numGestures = int(input("Enter Number of Gestures to Train On: "))
    for idx in range(numGestures):
        name = input("Enter Name for Gesture #" + str(idx + 1) + ": ")
        gestureNames.append(name)

def findDistance(pt1, pt2):
    dX = pt1.x - pt2.x
    dY = pt1.y - pt2.y
    dZ = pt1.z - pt2.z
    return math.sqrt(dX*dX + dY*dY + dZ*dZ)

def getTipDistances(handLm):
    lm = handLm.landmark
    wrist = lm[0]
    indexBase = lm[5]  
    scaleFactor = findDistance(wrist, indexBase)
    tips = [lm[4], lm[8], lm[12], lm[16], lm[20]]
    tipDistances = []
    for tip in tips:
        dist = findDistance(wrist, tip) / scaleFactor
        tipDistances.append(dist)
    for startTip in range(len(tips)):
        for endTip in range(startTip + 1, len(tips)):
            dist = findDistance(tips[startTip], tips[endTip]) / scaleFactor
            tipDistances.append(dist)
    return tipDistances

def calculateError(tipDist, trainDist):
    totalError = 0
    for idx in range(len(tipDist)):
        diff = tipDist[idx] - trainDist[idx]
        totalError += abs(diff)
    return totalError

def findGesture(tipLive, trainedData, maxAllowedError):
    bestMatch = "Unknown"
    lowestError = 999999999
    for gestureName, trainedDist in trainedData.items():
        error = calculateError(tipLive, trainedDist)
        if error < lowestError:
            lowestError = error
            bestMatch = gestureName
    if lowestError > maxAllowedError:
        return "Unknown"
    return bestMatch

# ====================================================================
# NEOPIXEL CONTROL (COLOR HELPER & ANIMATION DISPATCHER)
# ====================================================================
def hsv2rgb(h):
    """ Converts HSV float (0.0-1.0) to RGB integer tuple (0-255) """
    r, g, b = colorsys.hsv_to_rgb(h, 1, 1)
    return (int(r * 255), int(g * 255), int(b * 255))

def updateNeoPixels(gesture):
    """ Non-blocking LED pattern update based on the current detected gesture """
    global animation_step, lastAnimUpdate
    if not NEOPIXEL_ENABLED:
        return

    now = time.time()
    gestureClean = gesture.lower()
    pattern_matched = False

    # 1. FIST / CHARGE (Red pulse)
    if gestureClean == "fist":
        pulse = int(127 + 127 * math.sin(now * 5.0))
        strip.fill((pulse, 0, 0))
        strip.show()
        pattern_matched = True

    # 2. OPEN PALM / BEACON (Solid Bright White)
    if gestureClean == "palm":
        strip.fill((255, 255, 255))
        strip.show()
        pattern_matched = True

    # 3. PEACE SIGN / DUAL SPINNER (Rotating Blue/Cyan dots)
    if gestureClean == "peace":
        if now - lastAnimUpdate > 0.08:
            animation_step = (animation_step + 1) % NUM_LEDS
            lastAnimUpdate = now
        strip.fill((40, 0, 0))  # Background red
        pos1 = animation_step
        pos2 = (animation_step + (NUM_LEDS // 2)) % NUM_LEDS
        strip[pos1] = (0, 255, 0)   # Green spinner dot
        strip[pos2] = (0, 0, 255)   # Blue spinner dot
        strip.show()
        pattern_matched = True

    # 4. THUMBS UP / SUCCESS (Solid Green)
    if gestureClean == "thumbsup":
        strip.fill((0, 255, 0))
        strip.show()
        pattern_matched = True

    # 5. POINTING / CHASER (Radar sweep)
    if gestureClean in "point":
        if now - lastAnimUpdate > 0.06:
            animation_step = (animation_step + 1) % NUM_LEDS
            lastAnimUpdate = now
        strip.fill((255, 0, 0))
        strip[animation_step] = (0, 255, 0)
        strip.show()
        pattern_matched = True

    # 6. ROCK / FIRE (Sinusoidal Green-Cyan pulse)
    if gestureClean == "rock":
        step = int((now * 100) % 360)
        brightness = (math.sin(step * math.pi / 180) + 1) / 2
        g = int(255 * brightness)
        b = int(50 * brightness)
        strip.fill((0, g, b))
        strip.show()
        pattern_matched = True

    # 7. OK SIGN / RAINBOW WHEEL (Rotating RGB spectrum)
    if gestureClean in "ok":
        if now - lastAnimUpdate > 0.03:
            animation_step = (animation_step + 1) % 100
            lastAnimUpdate = now
        hue = animation_step / 100.0
        strip.fill(hsv2rgb(hue))
        strip.show()
        pattern_matched = True

    # DEFAULT / UNKNOWN (LEDs OFF)
    if not pattern_matched:
        strip.fill((0, 0, 0))
        strip.show()
# ====================================================================

W, H = 1280, 720
tStart = time.time()
fps = 15
RES = (W, H)

piCam = Picamera2(1)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate = 60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W * .01), int(H * .07))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W / 425)
fontScale = H * .002
fontColor = (0, 0, 255)

textFps = (int(W * .01), int(H * .07))
textGesture = (int(W * .01), int(H * .14))
textGesturePrompt = (int(W * .01), int(H * .21))

maxErrorThreshold = 3.5 

hands = mp.solutions.hands.Hands(
    model_complexity=0,
    min_detection_confidence=.5,
    min_tracking_confidence=.5,
    max_num_hands=2
)

trainIdx = 0

while True:
    deltaT = time.time() - tStart
    tStart = time.time()
    fps = fps * .95 + (1 / deltaT) * .05
    
    frame = piCam.capture_array()
    frame = cv2.flip(frame, -1)
    rgbFrame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    
    if results.multi_hand_landmarks:
        frameHeight, frameWidth, _ = frame.shape        
        for handLm in results.multi_hand_landmarks:
            for idx, landmark in enumerate(handLm.landmark):
                pixelX = int(landmark.x * frameWidth)
                pixelY = int(landmark.y * frameHeight)
                cv2.putText(
                    frame, str(idx), (pixelX, pixelY),
                    cv2.FONT_HERSHEY_SIMPLEX, .6 * fontScale, (255, 0, 0), fontThickness
                )
                
    if trainIdx < len(gestureNames):
        targetName = gestureNames[trainIdx]
        promptText = "show " + targetName + " & Press Space"
        cv2.putText(frame, promptText, textGesturePrompt, fontFace, fontScale, (0, 255, 255), fontThickness)

    detectGesture = "Unknown"
    if results.multi_hand_landmarks:
        liveDistances = getTipDistances(results.multi_hand_landmarks[0])
        detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)

    # ====================================================================
    # EXECUTE HARDWARE PATTERN UPDATE
    # ====================================================================
    updateNeoPixels(detectGesture)
    # ====================================================================

    labelText = "Gesture: " + detectGesture
    cv2.putText(frame, labelText, textGesture, fontFace, fontScale, (0, 255, 0), fontThickness)
    myText = "FPS: " + str(round(fps, 1))
    cv2.putText(frame, myText, textLowerLeft, fontFace, fontScale, fontColor, fontThickness)
    
    cv2.imshow("Camera", frame)
    #cv2.moveWindow("Camera", 0, 60)
    
    key = cv2.waitKey(1)
    
    # Save landmark measurements when Space is pressed
    if key == ord(' '):
        if results.multi_hand_landmarks:
            if trainIdx < len(gestureNames):
                targetHand = results.multi_hand_landmarks[0]
                targetName = gestureNames[trainIdx]
                trainedData[targetName] = getTipDistances(targetHand)
                saveTrainedData(DATA_FILE, trainedData)
                trainIdx += 1

    # 't' key allows adding a NEW gesture dynamically during runtime
    if key == ord('t'):
        newGestureName = input("Enter name for new gesture to train: ").strip()
        if newGestureName:
            gestureNames.append(newGestureName)

    # 'd' key deletes a gesture
    if key == ord('d'):
        delName = input("Enter name of gesture to remove: ").strip()
        if delName in trainedData:
            del trainedData[delName]
            saveTrainedData(DATA_FILE, trainedData)
            print("[INFO] Removed " + delName + " from trained data.")
            if delName in gestureNames:
                gestureNames.remove(delName)
        if delName not in trainedData:
            print("[WARNING] Gesture: " + delName + " not found in trained data.")
        trainIdx=min(trainIdx,len(gestureNames))
    if key == ord('q'):
        break

# Clean up hardware and exit
# ====================================================================
# CLEAR LEDS ON SHUTDOWN
# ====================================================================
if NEOPIXEL_ENABLED:
    strip.fill((0, 0, 0))
    strip.show()
# ====================================================================

cv2.destroyAllWindows()
saveTrainedData(DATA_FILE, trainedData)
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import math

import pickle

import os

import colorsys

# ====================================================================

# NEOPIXEL CONTROL INITIALIZATION (Adafruit neopixel_spi via SPI)

# ====================================================================

import board

import neopixel_spi as neopixel

NUM_LEDS = 12 # Set to match your NeoPixel ring size (e.g., 12)

PO = neopixel.GRB

# Initialize NeoPixel ring using SPI

spi = board.SPI()

strip = neopixel.NeoPixel_SPI(

spi,

NUM_LEDS,

pixel_order=PO,

auto_write=False

)

# Turn off all LEDs on startup

strip.fill(0)

strip.show()

NEOPIXEL_ENABLED = True

print("[INFO] NeoPixel SPI initialized successfully.")

# Animation state variables (non-blocking updates)

animation_step = 0

lastAnimUpdate = time.time()

# ====================================================================

DATA_FILE = "gestureNEOPIXEL.pkl"

# Function to save the trained gesture dictionary to a file using pickle

def saveTrainedData(filename, data):

with open(filename, 'wb') as f:

pickle.dump(data, f)

print("[INFO] Saved gesture data to " + str(filename))

# Function to load previously saved gesture data from a file

def loadTrainedData(filename):

if os.path.exists(filename):

with open(filename, 'rb') as f:

data = pickle.load(f)

print("[INFO] Loaded " + str(len(data)) + " gesture(s) from " + str(filename))

return data

return {}

# --- SETUP & INITIAL TRAINING ---

trainedData = loadTrainedData(DATA_FILE)

gestureNames = []

if not trainedData:

print("HAND GESTURE TRAINING SETUP")

numGestures = int(input("Enter Number of Gestures to Train On: "))

for idx in range(numGestures):

name = input("Enter Name for Gesture #" + str(idx + 1) + ": ")

gestureNames.append(name)

def findDistance(pt1, pt2):

dX = pt1.x - pt2.x

dY = pt1.y - pt2.y

dZ = pt1.z - pt2.z

return math.sqrt(dX*dX + dY*dY + dZ*dZ)

def getTipDistances(handLm):

lm = handLm.landmark

wrist = lm[0]

indexBase = lm[5]

scaleFactor = findDistance(wrist, indexBase)

tips = [lm[4], lm[8], lm[12], lm[16], lm[20]]

tipDistances = []

for tip in tips:

dist = findDistance(wrist, tip) / scaleFactor

tipDistances.append(dist)

for startTip in range(len(tips)):

for endTip in range(startTip + 1, len(tips)):

dist = findDistance(tips[startTip], tips[endTip]) / scaleFactor

tipDistances.append(dist)

return tipDistances

def calculateError(tipDist, trainDist):

totalError = 0

for idx in range(len(tipDist)):

diff = tipDist[idx] - trainDist[idx]

totalError += abs(diff)

return totalError

def findGesture(tipLive, trainedData, maxAllowedError):

bestMatch = "Unknown"

lowestError = 999999999

for gestureName, trainedDist in trainedData.items():

error = calculateError(tipLive, trainedDist)

if error < lowestError:

lowestError = error

bestMatch = gestureName

if lowestError > maxAllowedError:

return "Unknown"

return bestMatch

# ====================================================================

# NEOPIXEL CONTROL (COLOR HELPER & ANIMATION DISPATCHER)

# ====================================================================

def hsv2rgb(h):

""" Converts HSV float (0.0-1.0) to RGB integer tuple (0-255) """

r, g, b = colorsys.hsv_to_rgb(h, 1, 1)

return (int(r * 255), int(g * 255), int(b * 255))

def updateNeoPixels(gesture):

""" Non-blocking LED pattern update based on the current detected gesture """

global animation_step, lastAnimUpdate

if not NEOPIXEL_ENABLED:

return

now = time.time()

gestureClean = gesture.lower()

pattern_matched = False

# 1. FIST / CHARGE (Red pulse)

if gestureClean == "fist":

pulse = int(127 + 127 * math.sin(now * 5.0))

strip.fill((pulse, 0, 0))

strip.show()

pattern_matched = True

# 2. OPEN PALM / BEACON (Solid Bright White)

if gestureClean == "palm":

strip.fill((255, 255, 255))

strip.show()

pattern_matched = True

# 3. PEACE SIGN / DUAL SPINNER (Rotating Blue/Cyan dots)

if gestureClean == "peace":

if now - lastAnimUpdate > 0.08:

animation_step = (animation_step + 1) % NUM_LEDS

lastAnimUpdate = now

strip.fill((40, 0, 0)) # Background red

pos1 = animation_step

pos2 = (animation_step + (NUM_LEDS // 2)) % NUM_LEDS

strip[pos1] = (0, 255, 0) # Green spinner dot

strip[pos2] = (0, 0, 255) # Blue spinner dot

strip.show()

pattern_matched = True

# 4. THUMBS UP / SUCCESS (Solid Green)

if gestureClean == "thumbsup":

strip.fill((0, 255, 0))

strip.show()

pattern_matched = True

# 5. POINTING / CHASER (Radar sweep)

if gestureClean in "point":

if now - lastAnimUpdate > 0.06:

animation_step = (animation_step + 1) % NUM_LEDS

lastAnimUpdate = now

strip.fill((255, 0, 0))

strip[animation_step] = (0, 255, 0)

strip.show()

pattern_matched = True

# 6. ROCK / FIRE (Sinusoidal Green-Cyan pulse)

if gestureClean == "rock":

step = int((now * 100) % 360)

brightness = (math.sin(step * math.pi / 180) + 1) / 2

g = int(255 * brightness)

b = int(50 * brightness)

strip.fill((0, g, b))

strip.show()

pattern_matched = True

# 7. OK SIGN / RAINBOW WHEEL (Rotating RGB spectrum)

if gestureClean in "ok":

if now - lastAnimUpdate > 0.03:

animation_step = (animation_step + 1) % 100

lastAnimUpdate = now

hue = animation_step / 100.0

strip.fill(hsv2rgb(hue))

strip.show()

pattern_matched = True

# DEFAULT / UNKNOWN (LEDs OFF)

if not pattern_matched:

strip.fill((0, 0, 0))

strip.show()

# ====================================================================

W, H = 1280, 720

tStart = time.time()

fps = 15

RES = (W, H)

piCam = Picamera2(1)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate = 60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W * .01), int(H * .07))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W / 425)

fontScale = H * .002

fontColor = (0, 0, 255)

textFps = (int(W * .01), int(H * .07))

textGesture = (int(W * .01), int(H * .14))

textGesturePrompt = (int(W * .01), int(H * .21))

maxErrorThreshold = 3.5

hands = mp.solutions.hands.Hands(

model_complexity=0,

min_detection_confidence=.5,

min_tracking_confidence=.5,

max_num_hands=2

)

trainIdx = 0

while True:

deltaT = time.time() - tStart

tStart = time.time()

fps = fps * .95 + (1 / deltaT) * .05

frame = piCam.capture_array()

frame = cv2.flip(frame, -1)

rgbFrame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

frameHeight, frameWidth, _ = frame.shape

for handLm in results.multi_hand_landmarks:

for idx, landmark in enumerate(handLm.landmark):

pixelX = int(landmark.x * frameWidth)

pixelY = int(landmark.y * frameHeight)

cv2.putText(

frame, str(idx), (pixelX, pixelY),

cv2.FONT_HERSHEY_SIMPLEX, .6 * fontScale, (255, 0, 0), fontThickness

)

if trainIdx < len(gestureNames):

targetName = gestureNames[trainIdx]

promptText = "show " + targetName + " & Press Space"

cv2.putText(frame, promptText, textGesturePrompt, fontFace, fontScale, (0, 255, 255), fontThickness)

detectGesture = "Unknown"

if results.multi_hand_landmarks:

liveDistances = getTipDistances(results.multi_hand_landmarks[0])

detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)

# ====================================================================

# EXECUTE HARDWARE PATTERN UPDATE

# ====================================================================

updateNeoPixels(detectGesture)

# ====================================================================

labelText = "Gesture: " + detectGesture

cv2.putText(frame, labelText, textGesture, fontFace, fontScale, (0, 255, 0), fontThickness)

myText = "FPS: " + str(round(fps, 1))

cv2.putText(frame, myText, textLowerLeft, fontFace, fontScale, fontColor, fontThickness)

cv2.imshow("Camera", frame)

#cv2.moveWindow("Camera", 0, 60)

key = cv2.waitKey(1)

# Save landmark measurements when Space is pressed

if key == ord(' '):

if results.multi_hand_landmarks:

if trainIdx < len(gestureNames):

targetHand = results.multi_hand_landmarks[0]

targetName = gestureNames[trainIdx]

trainedData[targetName] = getTipDistances(targetHand)

saveTrainedData(DATA_FILE, trainedData)

trainIdx += 1

# 't' key allows adding a NEW gesture dynamically during runtime

if key == ord('t'):

newGestureName = input("Enter name for new gesture to train: ").strip()

if newGestureName:

gestureNames.append(newGestureName)

# 'd' key deletes a gesture

if key == ord('d'):

delName = input("Enter name of gesture to remove: ").strip()

if delName in trainedData:

del trainedData[delName]

saveTrainedData(DATA_FILE, trainedData)

print("[INFO] Removed " + delName + " from trained data.")

if delName in gestureNames:

gestureNames.remove(delName)

if delName not in trainedData:

print("[WARNING] Gesture: " + delName + " not found in trained data.")

trainIdx=min(trainIdx,len(gestureNames))

if key == ord('q'):

break

# Clean up hardware and exit

# ====================================================================

# CLEAR LEDS ON SHUTDOWN

# ====================================================================

if NEOPIXEL_ENABLED:

strip.fill((0, 0, 0))

strip.show()

# ====================================================================

cv2.destroyAllWindows()

saveTrainedData(DATA_FILE, trainedData)

print('Program Terminated')

Homework & Next Steps

Try setting up the circuit and running the code on your Raspberry Pi 5. Observe how the frame rate behaves when driving both the camera preview, the hand-tracking model, and the I2C OLED display simultaneously. For your homework, extend this script to include a dynamic color slider or gesture-based brightness control!

Technology Tutorials

AI on the Edge LESSON 58: Object Detection With Speech to Text Voice Control on Raspberry Pi 5

Understanding the Program Architecture

1. Voice-Driven Input via Background Threading

2. Thread-Safe Command Queuing

3. High-Speed OpenCV and MediaPipe Vision Loop

4. Proportional Servo Feedback Control

Hardware Requirements

Homework Assignment for Next Week

AI on the Edge LESSON 57: Detecting and Tracking Objects with MediaPipe

What You Will Learn in Lesson 57

Hardware Requirements

How the Proportional Tracking Control Works

Homework Assignment

General Artificial Intelligence & Edge Computing Concepts

What is Edge AI?

Key Advantages of Local Inference:

Proportional-Integral-Derivative Control Concepts:

AI on the Edge LESSON 56: Identifying Object of Interest in Target Zone with Mediapipe

AI on the Edge LESSON 56: Identifying Object of Interest in Target Zone with MediaPipe

What We Are Doing Today

Homework / Challenge Assignment

Discussion & Community

General Knowledge / Context

AI on the Edge LESSON 55: Object Detection on the Raspberry Pi 5 With MediaPipe

What We Are Doing in Lesson 55

Key Takeaways & Concepts Covered

What You Will Need

Homework Assignment

AI on the Edge LESSON 54: Custom Hand Gesture Recognition & Dynamic NeoPixel Control

System Architecture Overview

Gesture-to-Hardware Animation Mapping

Interactive Keyboard Controls

Hardware Wiring Reference

Homework Challenge

Homework & Next Steps

Making The World a Better Place One High Tech Project at a Time. Enjoy!