How to Build a Simple Computer Vision Texting App Using Twilio

There are many use cases in which it could be beneficial to have automated text messages sent that contain data obtained from computer vision. Perhaps you’d like to be notified whenever a person or animal walks into your yard or house (by using object detection), or when your kids appear to be fighting (by using pose estimation), or any other number of scenarios — the possibilities are endless! In this tutorial, we’ll show you how easy it is to accomplish this task by using a very basic example of a computer vision app that can detect a chair, and which sends a text message letting the user know when a chair has been detected.

Requirements:

• Create an alwaysAI account
• Install alwaysAI onto your development computer
• Create a Twilio account and get a Twilio phone number and an auth token
• Clone the repo - https://github.com/alwaysai/alwaysai-twilio-sms

*Note: Following this guide will be simplest if you run both servers on the same machine (as opposed to running one on your laptop and one on an edge device), since you will be targeting localhost ports.

Brief overview: We're going to run 2 separate servers that send data to each other using Socket.io. We’ve modified the alwaysAI realtime_object_detector starter app so that it contains the Socket.io client, and can send data to our express server (which contains the Socket.io server) as it runs. Once we capture that data in our express server, we will send a text message containing that data using the Twilio API.

Step 1: server.js

After getting set up and cloning the repository, run these commands in your terminal to navigate to the directory and install the necessary npm modules:

$ cd alwaysai-twilio-sms/server
$ sudo npm install -g nodemon #(nodemon is a utility that will monitor for any changes in your source and automatically restart your server.)
$ npm install #(installs all the npm modules needed)

The express server.js file is where all of our texting logic is kept, as well as our Socket.io server.

Starting on line 23 in the server.js file, you can see a socket event called ‘labels’ that will trigger when an event named ‘labels’ is emitted from the realtime_object_detector application.

Essentially, there will be a method in the realtime_object_detector application that will send messages to the event with the same name in our express server.js application.

io.on('connection', function (socket) {
  socket.on('labels', function (data) {
    
    console.log(data)
    
    // the twilio text only gets sent if the trigger variable is set to true
    if (trigger){
      client.messages.create({
        body: 'You see a chair',
        from: '+12000000000',
        to: '+12000000000'
      }).then(message => console.log(message.sid));

      trigger = false

      // this timeout method is defined previously. It will redefine the trigger variable as true when 20 seconds pass 
      timeout()
    }

  });
});

Once that event is triggered, our next step will be to execute the texting functionality. In this case, we can execute without checking that the data is being sent, since we’ve already added a conditional statement in the realtime_object_detector to check for this, but you can edit this or another check if you’d like.

For example:

if (data.response === 'chair'){
  //do something
}

The texting functionality is done using the Twilio modules, which allow us to send a text simply by adding an Auth Token, our Twilio number and a desired target number.

Edit the attributes called below and their respective values (replace the "from" value with your new Twilio number, and the "to" value with your own phone number), and update the message that you want to send:

const accountSid = '';
const authToken = '';

{
  body: 'You see a chair',
  from: '+12000000000',
  to: '+12000000000'
}

And run this command in your terminal to start the server:

$ nodemon ./server.js  #(starts the server and rebuilds if there are any changes made in the server.js file)

If you have something like the output below you can move on to the next step:

[nodemon] starting `node ./server.js`
server is running on port 3001

Step 2: app.py

Next, run this command to navigate to the app directory:

$ cd alwaysai-twilio-sms/alwaysai-starter-apps/realtime_object_detector

Now we're in an alwaysAI starter app called the realtime_object_detector. The only difference between this app and the original (that you would download from alwaysai.co) is that this one has a simple Socket.io client that will connect to our express server.

In the app.py file there is a class called ClientSocket that initializes a Socket.io client. Change the URL if needed in sio.connect('http://localhost:3001') — this is where you establish which server will receive the information. If you are deploying to a remote device, you will need to find and use your computer's IP address instead — e.g., sio.connect('http://192.168.0.10/3001').

class ClientSocket():
    sio = socketio.Client()

    @sio.event
    def connect():
        print('connection established')

    @sio.event
    def labels(self, data):
        self.sio.emit('labels', {'response': data})

    @sio.event
    def disconnect():
        print('disconnected from server')

    sio.connect('http://localhost:3001')

Remember when we talked about the ‘labels’ event? This is where we create that:

    def labels(self, data):
        self.sio.emit('labels', {'response': data})

The first parameter in the emit method is the event name, and since we called our event ‘labels’ over on the express side, we’ll need to use that name as our emit event name as well.

Now that we have both of our applications connected via Socket.io, we can add some restrictions on when to send data over our events. We could send all of the data being processed, but that might not be too useful.

On line 21 we see that there is a variable named ‘target’ that holds a string of our desired label target — in our case this is 'chair'. If you change this target to a different label (e.g., from 'chair' to 'person'), just remember to change the message text in the server.js file accordingly — e.g., 'you see a person.'

    target = 'chair'

Now within the prediction loop of our application, we will check if the prediction.label matches our target.

for prediction in results.predictions:
                    if target == prediction.label:
                      client_socket.labels(prediction.label)


                    text.append("{}: {:2.2f}%".format(
                        prediction.label, prediction.confidence * 100))

                streamer.send_data(frame, text)

                fps.update()

If it does, then it will send data via the socket event named ‘label.’ Basically, if the prediction is a label with ‘chair,’ the socket event will trigger and send data over to our express server.

Step 3: testing the app

If everything looks good, open a new terminal window (while still running the server in the other one) and run:

$ aai app install
$ aai app start

And that’s it! Once your alwaysAI application detects a chair, it should send you a text.

If there are any errors, make sure to read the logs on both ends — the express side and the alwaysAI side.

Conclusion

We’ve walked through a simple example of extending an alwaysAI starter app to incorporate Twilio, which demonstrates how easy it is to integrate the alwaysAI computer vision platform with any existing service or application. We can’t wait to see what you build with alwaysAI!