Create Your Own Contraband Detector

April 14, 2020 by

Lila Mullany

Many people are now working or learning from home, introducing the new issue of enforcing professionalism and academic honesty by remote. Whether you are trying to prevent students from cheating on tests, want to double check that your kids aren’t just playing on their phones, or maybe you want to stop yourself from checking social media while you’re supposed to be working, computer vision can help. In this tutorial, I’ll show how you can create your own contraband object alert system using computer vision.

To complete the tutorial, you must have:

An alwaysAI account (it’s free!)
alwaysAI set up on your machine (also free)
A text editor such as sublime or an IDE such as PyCharm, both of which offer free versions, or whatever else you prefer to code in

Please see the alwaysAI blog and tutorials for more background on computer vision, developing models, how to change models, and more.

All of the code from this tutorial is available on GitHub.

Let’s get started!

After you have your free account and have set up your developer environment, download the starter apps; do so using this link before proceeding with the rest of the tutorial.

There will be two main parts to this tutorial:

Creation of the main app
Creation of the utility class to store and access a detection log

To reduce overhead due to inference time as well as CPU usage, we will use a correlation tracker in this app. In this case, we will only look at every 50 frames for the target object, which is sufficient for our purposes, as we do not need to track the objects in every frame, just capture the occurrence.

We will be using two object detection models in order to maximize the amount of relevant detections. The two models we are using are ‘alwaysai/ssd_mobilenet_v2_oidv4’ and ‘alwaysai/ssd_inception_v2_coco_2018_01_28’. The first model, ‘alwaysai/ssd_mobilenet_v2_oidv4’, has a larger library of relevant labels and includes items such as ‘headphones’. The second model, ‘alwaysai/ssd_inception_v2_coco_2018_01_28’, detects certain objects, such as ‘cell phones’ and ‘books’, more readily and consistently. You can see the original post on using two detectors simultaneously for more information.

Part I - Creation of app.py

Since we’re using an object tracker, we will modify the ‘detector_tracker’ starter app; cd into that directory and open up app.py.
Replace the lines declaring ‘obj_detect’, the subsequent call to ‘obj_detect.load’, and the print statements that follow with the following code:

# to test an additional model, add it to the list rbelow:
models = [“alwaysai/ssd_mobilenet_v2_oidv4”,”alwaysai/ssd_inception_v2_coco_2018_01_28"]

# include any labels that you wish to detect from any models 
detected_contraband = [“Pen”, “cell phone”, “backpack”, “book”, “Book”, “Ring binder”, “Headphones”, “Calculator”, “Mobile phone”, “Telephone”, “Microphone”, “Ipod”, “Remote control”]

# load all the models (creates a new object detector for each model)
detectors = []
for model in models:
    # start up a first object detection model
    obj_detect = edgeiq.ObjectDetection(model)
    obj_detect.load(engine=edgeiq.Engine.DNN)

    # track generated object detectors by storing them in detectors
    detectors.append(obj_detect)

    # print the details of each model to the console
    print(“Model:\n{}\n”.format(obj_detect.model_id))
    print(“Engine: {}”.format(obj_detect.engine))
    print(“Accelerator: {}\n”.format(obj_detect.accelerator))
    print(“Labels:\n{}\n”.format(obj_detect.labels))

The declaration of models builds a list of all models we would like to include. The declaration of ‘detected_contraband’ specifies which labels from those models we want to check the video stream for. We then build a list of detection objects, stored in ‘detectors’ and perform the initialization, loading, and printing of the details from the original app on each detection object.

3. Instantiate a contraband tracker by adding the following line underneath the declaration of the ‘fps’ variable.

contraband_summary = ContrabandSummary()

4. Additionally, add the following declaration to the top of the code

from contraband_summary import ContrabandSummary

Don’t worry about the ContrabandSummary code just yet, we’ll come back to that in the second part of the tutorial.

5. Everywhere in the code, change references to the variable called ‘predictions’ to instead refer to ‘predictions_to_markup’. This is a small distinction, but we won’t be marking up every prediction and this reinforces that.

6. Add a declaration of a variable named ‘text’ and initialize it to be a list containing an empty string. The results from steps 5 and 6 should look like this:

while True:
    frame = video_stream.read()

    predictions_to_markup = []

    text = [“”]
 
    # only analyze every ‘detect_period’ frame (i.e. every 50th)
    if frame_idx % detect_period == 0:

7. Remove the declarations of the ‘text’ variable from within the ‘if’ statement.

8. Replace all of the code inside the ‘for prediction in predictions’ loop with the following:

if (prediction.label.strip() in detected_contraband):
    contraband_summary.contraband_alert(prediction.label, frame)
    predictions_to_markup.append(prediction)
    tracker.start(frame, prediction)

This code only takes action if the detected object is in the list of contraband we defined in the beginning. If it finds a contraband item, it updates with contraband_summary with a call to contraband_alert() and adds the prediction to the ‘predictions_to_markup’. Finally, it starts tracking these new detections.

9. Finally, beneath the initialization of the ‘frame’ variable, add the following code to update the ‘text’ variable:

text =contraband_summary.get_contraband_string()

That’s all we need for the first part of this app! Now we will create the contraband_summary utility class.

Part II - Creation of contraband_summary.py

This separate utility class is used to abstract away how we handle contraband detections. By separating this logic away from the main app, we can change how we handle detections at any time without breaking the main application. For now, we simply store all detections and generate a log string that can be used to print to the console or returned to a calling class. Later, we could change contraband_summary to send an sms alert or save the images to disk and we wouldn’t need to alter app.py to do so.

Create a new file called ‘contraband_summary.py’, either by using ‘touch contraband_summary.py’ from the command line or by creating a new file using the file browser, inside the same directory as app.py.
Include the time module by adding the following line to the very top of the file:

import time

3. Declare a class called ‘ContrabandSummary’:

class ContrabandSummary:

4. Inside this class declaration, create a constructor for the ContrabandSummary class that initializes an empty list, which we will use to store contraband detections:

def __init__(self):
    self.contraband_detections = []

5. Create a ‘contraband_alert’ method. This method takes in a string, which is intended to be the label of the object, and the numpy array of the video frame. It calculates the local time and appends a tuple containing the string, the time, and the frame to the ‘contraband_detections’ list. The method is implemented in the following way:

def contraband_alert(self, contraband, frame):
    detect_time = time.localtime()
    self.contraband_detections.append((contraband, detect_time, frame))
    items = self.get_contraband_string()

    print(*items)

Notice the method calls another method in the ContrabandSummary class: get_contraband_string(). By separating the functionality of generating the log, we can change the way this log is presented in the future and we also gain the ability to call these two methods independently.

6. The get_contraband_string() method builds a log of all the detections of contraband items. We just want to know the item that was detected and the time that the detection occurred. Implement this method by copying the following code into ContrabandSummary class below the previous method:

def get_contraband_string(self):
    contraband_string = [“”]

    for contraband, detect_time, frame in self.contraband_detections:
        string_time = time.strftime(‘%Y-%m-%d %H:%M:%S’, detect_time)
        contraband_string.append(contraband + “ detected at “ + string_time + “\n”)

    return contraband_string

NOTE: we don’t do anything with the collected numpy frame in this example, but you can see how this implementation offers the flexibility to change how to store or manipulate the data in however fashion best suits your needs. You could write the image to a folder, or send it to another application along with the text. You could also alter the alert function to relay the contraband detection in a text message.

That’s it!

Now, to see your app in action, first build the app by typing into the command line: