The Story of The Little Robot that Could

Dec 12, 2023

9 min read

My Introduction to Computer Vision

In 2015, my journey into the realm of Artificial Intelligence began with a captivating challenge at Western Kentucky University. Our intermediate software group decided to take on a project that was not just ambitious but also a foray into uncharted territory for us. The mission? To create two cooperative autonomous robots, each equipped with a camera, designed to work in tandem. Their task was to navigate a specific area and collaboratively identify three distinctively colored tennis balls: blue, yellow, and pink.

This project was more than just about building robots; it was a deep dive into three pivotal technologies that were new to us at the time:

Robotics and Autonomous Movement: The physical manifestation of our project, robotics involved not just the assembly of the robots but also understanding their movement, control mechanisms, and how they interact with their environment.

Computer Vision: This was the cornerstone of our project, enabling the robots to 'see' and identify the colored tennis balls. We ventured into the complexities of image processing and object detection, learning how to interpret and respond to visual data.

Custom Client-Server Communications: Essential for the cooperative aspect of our robots, we developed a system for them to communicate with each other. This involved setting up a network, managing data transmission, and ensuring that both robots shared information effectively to achieve their common goal.

Our Creation

Pause for a second to appreciate the masterpiece we've created: a robot armed with proximity sensors, DC motorized wheels, a trusty mobile power bank, an Arduino Uno, a Raspberry Pi complete with a camera, and an 'indestructible' chassis crafted from the finest materials known to DIY enthusiasts – acrylic, the ever-reliable duct tape, and good old cardboard. Jokes aside, there's a real sense of pride in how we pulled off this project with the resources at hand.

Its important to note that when we had less than 12 weeks and a plethora of unknown technologies to explore, we wanted to get something functional for prototyping and come back to it if we had time.

Autonomous Movement

One of our initial hurdles was achieving autonomous movement. After delving into proximity sensors and Arduino coding, we managed to piece together a traversal method that was more functional than intelligent. The real challenge, however, lay in the quality of our sensors. We encountered over five sensors that were defective right from the start. The ones we finally chose had a limited detection range, only sensing objects within about 3 inches in front of the robot, and that too, inconsistently. Over time, we grew fond of its quirky imperfections, which you'll definitely notice in the robot's amusing movements in our final demonstration video.

Here is the movement script we utilized developed by Firas Alsolaim and Moteab Alshrimi:

from nanpy import (ArduinoApi, SerialManager)
from time import sleep
from nanpy.measure import Measure
def main():
    try:
        connection = SerialManager()
        a = ArduinoApi(connection=connection)
        m = Measure(connection=connection)
    except:
        print("Faild to connect to Arduino Firas")

    # setup the pinMode as is we were in the arduino IDE

    a.pinMode(2, a.OUTPUT)
    a.pinMode(3, a.OUTPUT)
    a.pinMode(4, a.OUTPUT)
    a.pinMode(5, a.OUTPUT)

    try:
        while True:
            forward(a)
            if distance(m,6,7) < 30:
                print("right")
                reverse(a)
                right(a)
            elif distance(m,8,9) < 30:
                print("left")
                reverse(a)
                left(a)
             
    except:
        print("Faild2 ")
        a.digitalWrite(2, a.LOW)
        a.digitalWrite(3, a.LOW)
        a.digitalWrite(4, a.LOW)
        a.digitalWrite(5, a.LOW)

def reverse(a):
    a.digitalWrite(2, a.LOW)
    a.digitalWrite(3, a.HIGH)
    a.digitalWrite(4, a.HIGH)
    a.digitalWrite(5, a.LOW)
    sleep(0.2)
    stop(a)

def forward(a):
    a.digitalWrite(2, a.HIGH)
    a.digitalWrite(3, a.LOW)
    a.digitalWrite(4, a.LOW)
    a.digitalWrite(5, a.HIGH)
    
def right(a):
    a.digitalWrite(2, a.LOW)
    a.digitalWrite(3, a.HIGH)
    a.digitalWrite(4, a.LOW)
    a.digitalWrite(5, a.HIGH)
    sleep(0.2)
    stop(a)

def left(a):
    a.digitalWrite(2, a.HIGH)
    a.digitalWrite(3, a.LOW)
    a.digitalWrite(4, a.HIGH)
    a.digitalWrite(5, a.LOW)
    sleep(0.2)
    stop(a)
    
def stop(a):
    a.digitalWrite(2, a.LOW)
    a.digitalWrite(3, a.LOW)
    a.digitalWrite(4, a.LOW)
    a.digitalWrite(5, a.LOW)

def distance(m,trg,ech):
    distance = m.getMeasure(trg,ech)
    sleep(0.3)
    if(distance < 0):
        distance = distance * -1
        print(distance)
    return distance

main()

A demo of our first successful movement functions:

Object Detection (OpenCV)

The second, and arguably the most thrilling, challenge of our project was figuring out how our robots would detect three differently colored tennis balls. We turned to OpenCV for this task. Initially, we ambitiously planned to capture and process images from the Raspberry Pi camera at a rate of about 20-30 frames per second. However, we quickly discovered that the Raspberry Pi's processing speed, particularly in its interaction with the camera, was not up to the task for such a high frame rate. So, we scaled back our ambitions and aimed for a more achievable goal: processing one image per second.

With this approach, we could capture the view in front of the robot, but the big question remained: how do we detect the tennis balls, and more importantly, how do we identify each color separately? The solution we landed on was a multithreaded approach. We set up three separate threads to process the input image asynchronously, with each thread dedicated to one of the colors - blue, yellow, and pink.

Our initial strategy involved using masks over the hue and saturation values (HSV) of these colors. This technique allowed us to isolate and identify objects in the image that matched our specified color criteria (as demonstrated below).

As evident from the image, the mask didn't produce a perfect circle around the tennis balls. This imperfection was due to factors like shadows on the ball, its black logo, and the distinctive grooves of the tennis ball's rubber surface. These irregularities made our initial idea of applying circle detection on the mask unfeasible. Additionally, we faced another challenge: objects within the same HSV range as our defined colors were interfering with our mask, leading to inaccurate detections.

This is where Tyler Carrico stepped in with an ingenious solution: blob detection. By shifting our focus from circle detection to blob detection, we were able to more effectively identify and isolate the tennis balls despite the imperfections and obstructions in the mask.

import cv2
import numpy
import glob

#declare class
class DetectCV:
	def __init__(self):
		pass

	#used to detect obj
	def detect(self, c): 
		shape ="unidentified"
		perimeter = cv2.arcLength(c,True)
		approx = cv2.approxPolyDP(c, 0.04 * perimeter, True)

	# if there are more than 5 sides object is ball
		if len(approx) > 5:
			shape = "ball"
 
		# return the name of the ball
		return shape

We then utilized this function in our main search:

def search():
    global currentChecklist
    camera = PiCamera()
    
    def hsvThresholding(color,loH,loS,loV,upH,upS,upV,imageName):
        #start_time = time.time()
        image = cv2.imread(imageName)
        resized = imutils.resize(image, width=300)
        hsv = cv2.cvtColor(resized, cv2.COLOR_BGR2HSV)
        
        lower = np.array([loH,loS,loV])
        upper = np.array([upH,upS,upV])

        blur = cv2.GaussianBlur(hsv, (5, 5), 0)


        # mask the HSV image to get only  color searched for
        mask = cv2.inRange(blur, lower, upper)
        res = cv2.bitwise_and(resized,resized, mask= mask)


        #morphological transformations
        kernel = np.ones((5,5),np.uint8)
        mask = cv2.dilate(res, kernel, iterations=2)
        mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, kernel)
        mask = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)
        mask = cv2.erode(mask, kernel, iterations=2)


        #Create SimpleBlobdetector then create parameters
        params = cv2.SimpleBlobDetector_Params()


        #Circularity parameters
        params.filterByCircularity = True
        params.minCircularity = 0.1


        #Convexity parameters
        params.filterByConvexity = True
        params.minConvexity = 0.5

        detector = cv2.SimpleBlobDetector_create(params)
       
        #Detect blobs using detector and reverse mask
        reversemask=255-mask
        keypoints = detector.detect(reversemask)

        #end_time = time.time()
       # print (end_time - start_time)
        
        if keypoints:
            print "Found "+color+" ball"
            sock.send(color)
        else:
            print "not found"

        
    while True:
        ##Test case: image = 'pink1.jpg'
        time.sleep(.5)
        camera.capture('currentFrame.jpg')
        image = 'currentFrame.jpg'
        print "---search---"
        if currentChecklist[0] == 0:
            yellowImage = deepcopy(image)
            yellow = threading.Thread(name='yellow',target=hsvThresholding, args=('yellow',28,200,175,31,255,215,yellowImage))
            yellow.setDaemon(True)
            yellow.start()
        if currentChecklist[1] == 0:
            pinkImage = deepcopy(image)
            pink = threading.Thread(name='pink',target=hsvThresholding, args=('pink',1,0,50,4,255,255,pinkImage))
            pink.setDaemon(True)
            pink.start()
        if currentChecklist[2] == 0:
            blueImage = deepcopy(image)
            blue = threading.Thread(name='blue',target=hsvThresholding, args=('blue',93,0,50,105,255,255,blueImage))
            blue.setDaemon(True)
            blue.start()

Here is a quick demonstration of the tennis ball detection with the three different colors:

Client-Server Communication

The last hurdle we faced was perhaps the most daunting: enabling communication between the two robots. None of us had prior experience in building client-server communications, so this task was particularly challenging. After some deliberation, we decided on a setup involving a Python server script and a Python client script.

The server script was hosted on my MacBook during runtime, acting as the central hub for our operation. Each Raspberry Pi was equipped with the client script, which we accessed remotely. This setup allowed the server to manage client connections, keep track of the global search records for the tennis balls, and distribute updated search lists to the clients whenever there was a change.

One of the most exciting features of our server was its ability to announce when all tennis balls had been located and then shut down its clients. We designed the server with scalability in mind; it was built to handle more than the two clients we used in our experiment. This scalability was crucial in demonstrating that our cooperative distributed search approach could be expanded to include more agents if necessary.

import socket
import threading
import array
import pickle

globalChecklist = [0,0,0,0,0]
clients = set()
clients_lock = threading.Lock()

class ThreadedServer(object):
    
    def __init__(self, host, port):
        self.host = host
        self.port = port
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        self.sock.bind((self.host, self.port))

    #Continuously listening for incoming connection requests from Client.py
    #Accepts connection and adds the client to the set of all current clients
    #Calls listenToClient()
    def listen(self):
        self.sock.listen(5)
        while True:
            client, address = self.sock.accept()
            with clients_lock:
                clients.add(client)
            print client
            threading.Thread(target = self.listenToClient,args = (client,address)).start()

    #Sends the current checklist to the newly connected client
    #Listens for a message from the client , int (1,2,3,4,5)
    #Checks the number recieved off of the globalChecklist
    #Sends the updated checklist to every client 
    def listenToClient(self, client, address):
        size = 1024
        client.send("GLOBAL CHECKLIST")
        while not globalChecklist == [1,1,1,1,1]:
            try:
                data = client.recv(size)
                if data:
                    response = data
                    data = int(data)
                    if data == 1:
                        globalChecklist[0] = 1
                    elif data == 2:
                        globalChecklist[1] = 1
                    elif data == 3:
                        globalChecklist[2] = 1
                    elif data == 4:
                        globalChecklist[3] = 1
                    elif data == 5:
                        globalChecklist[4] = 1
                    else:
                        print "INVALID INPUT"
                    print response
                    print globalChecklist

                    if globalChecklist == [1,1,1,1,1]:
                        client.send("-1")
                        print "Closing Client Sockets..."
                        
                    else:
                        sendableChecklist = pickle.dumps(globalChecklist)
                        with clients_lock:
                            for c in clients:
                                c.sendall(sendableChecklist)
                else:
                    raise error('Client disconnected')

            except:
                client.close()
                return False

if __name__ == "__main__":
    port_num = 8080;
    ThreadedServer('',port_num).listen()

On the client side, each robot had a specific set of responsibilities tied to the global search records. The primary task was to determine whether the tennis balls detected by the object detection script (as mentioned earlier) were already found or not. If a robot discovered a new tennis ball, it communicated this information to the server. Conversely, if the tennis ball had already been logged as found in the global records, the client would bypass the call to the server. This system ensured efficient communication and prevented redundancy in the search process.

import socket
import pickle

#create a socket and connect to port 8080 on host network
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = 'localhost'
port = 8080;
sock.connect((host,port))

#This sock.recv(1024) recieves the current global checklist upon connection
#This solves reconnection issues
currentChecklist = sock.recv(1024)


while True:
    # data will be replaced with DetectDriver()'s output on which ball is found (1-5)
    data = raw_input("message: ")
    sock.send(data)
    # recieved = updated checklist unless the search is over
    # then recieved = "-1"
    recieved = sock.recv(1024)
    if (recieved == "-1"):
        # here is where all shutdown commands will be for other functions
        sock.close()
        print "Socket closed"
        exit()
    else:
        currentChecklist = pickle.loads(recieved)
        print currentChecklist

##IMPORTANT VARIABLES
        # currentChecklist - will be referenced to determine which colors to threshold
        # data - input 1-5 from DetectDriver

Below is a demonstration of our sever-client connections:

If we had more time:

We accomplished a lot with this project, but there is a laundry list of ideas we had to enhance the results, such as:

Improved movement - Move toward any detected colors that have not been found instead of an "always-forward" approach. This would include mapping the movement of the robot so it doesnt search the same path twice.
Improved camera position and quality - Lower the camera and upgrade it to a camera of higher megapixel to get more accurate image detection
Better end-of-search interaction - Instead of disconnecting the client, bring the robots back to a "home" position, then stop movement until another search is initialized
Implement a UI - Better UX for the server side results
Improve the image detection - We believe there are better methods to handle detection. With more time we may have utilized a technology like TensorFlow to create classification models of objects
Create a database of objects - Extrapolate this idea to other objects instead of just tennis balls
Construct a better robot/drone - This one is a given, better robot would lead to more consistent performance and movement

The Final Result

This project was a rollercoaster, and let me tell you, it was far from perfect. We stumbled through countless mistakes and probably wandered down some less-than-efficient paths. But hey, that's not what this story is about. This blog post is a celebration of a project that ignited my passion for AI. It's about how my team and I bravely dove into uncharted waters!

We embraced every imperfection, every 'oops' moment, and every 'why isn't this working?' with a smile. Because in the end, what we gained was so much more than just a finished project. We walked away with a sense of triumph, knowing we'd tackled something we once thought impossible.

And here's the thing: if you're ever staring at a project that makes you feel a bit out of your depth, jump in! You'll never know what amazing things you'll learn or how much fun you'll have figuring it all out. Embrace the chaos, love the learning curve, and who knows? You might just find a new passion, just like I did with AI.

Enjoy the beauty that is our Autonomous Search Robot: