How to Use Your Alexa to Control Your Home Devices, Water Your Plants or Even Feed Your Cat

By day I'm a general partner at Initialized Capital helping founders and investing in startups. But I've never given up my hunger to build things and hack on projects in my spare time. So in that spirit, I wanted to share a quick tutorial:

In this tutorial, we'll walk through the basics of how to set up an Alexa Skill to control a Raspberry Pi or Arduino. Instead of having to water your plants with a button or turn off your lights with an app you just tell Alexa what to do and it will relay to your device. There’s a whole host of robot minions you can build (more tutorials coming soon!) but with this one I’ll demonstrate the very basic functionality and how to bridge Arduino and Raspberry Pi with Alexa. The tutorial is intended for engineers and should take about an hour.

Both the Arduino and Raspberry Pi have general-purpose input/output pins, or GPIO for short. GPIO pins are used to connect with other physical devices, such as lights or motors. These can be toggled between On (high) or Off (low states). In this tutorial, we’ll connect the GPIO pin to a relay with an LED light that will helps us verify that our code is working. You can use the GPIO for binary toggling or expand the code to any other functionality afforded by Arduino and Raspberry Pi (playing music, watering plants, robot army, etc.).

To send commands to our Arduino or Raspberry Pi, we’ll combine an Alexa Skill built with AWS Lambda with PubNub. Here’s an outline of how it works:

  1. Alexa listens for a voice command

  2. Alexa pings a AWS Lambda end point with the intent/command

  3. Lambda publishes the command to a dedicated PubNub channel

  4. The Arduino/Raspberry Pi is subscribed to the PubNub channel and is “listening” for any new commands

  5. The Raspberry Pi or Arduino receives command from PubNub and toggles GPIO Pin

0) Set up

In order to have a communication bridge between the Amazon Lambda function and the Raspberry Pi/Arduino we’ll need a shared communication channel that both can publish and subscribe to. Pubnub provides this service for free and makes it easy to use.

You’ll need to create a PubNub account and a new PubNub application. Let’s call application “GPIO Control”. Click into GPIO “Control” and you’ll see a “Demokey” set. Make note of the “Publish Key” and “Subscribe Key” we’ll need it shortly.

Screen Shot 2017-08-29 at 94352 PM copypng

1) Set Up An Amazon Alexa Skill

This is a detailed overview of setting up an Amazon Skill. If you’ve done this before, just copy the intents, and sample utterances, and skip to the lambda section. Remember to record your app id and keep it handy.

High level what we’re doing here is telling Alexa what phrase (“invocation”) we want to activate our app (“skill”) and then what commands (“intents”) to listen for. We also specify what end point it should ping when it hears an intent, in our case Lambda.

Make a new app

Go to https://developer.amazon.com/edw/home.html#/ click on “Alexa” on the top navigation, then select “Alexa Skills Kit” and on the “Add a New Skill”. Now we’ll give Amazon some information about the app, don’t worry about this as you can change all of it later:

  • Leave skill type as “Custom Interaction Model”

  • Type in “GPIO Control” for the name

  • Type in “Control” for the invocation (this is what we’ll say to activate this app)

  • Leave the other fields at “no” since we’re not using any of that

Click “save” and then the application ID will be generated. Keep this handy. We will need it for the Lambda function.

Screen Shot 2017-08-29 at 94555 PM copypng

Add Intents

On the next page you should see Intents. This is the actions your skill can take. In our case we want to be able to turn off and on particular GPIOs

For intents put in:

There’s a new schema builder available now, but to keep things easy let’s just use simple JSON. Now let’s enumerate what can be given to our “turnOn” and “turnOff” commands:

Under “Enter Type” put in:

LIST_OF_GPIOS

In the “Enter Values” textbox put in our individual GPIO pins (you can expand this to however many you have, I’m only using #5 in this example):

2

3

4

5

Click “add”.

This lists the types of inputs (in our case pin numbers) you can give to your “TurnOn” / “TurnOff” command. You can add more GPIO pins if you have more. For example, if you were selecting a type of tea this could be the list of teas or if you're controlling.

Define Sample Utterances

Now you’ll need to define what you say in order to turn the GPIO pins on or off. In the textbox under “Sample Utterances” put in:

TurnOn Turn on {GPIO}

TurnOn Turn {GPIO} on

TurnOff Turn off {GPIO}

TurnOff Turn {GPIO} off

AllOff Turn everything off

AllOn Turn everything on

AllOn All on

Feel free to add more. The format here is:

*your intent* (must be one of those you added above) *how to invoke it*

In our example  “All On” and “Turn everything on” both invoke the same intent (AllOn). To turn on pin 5 - you would say “Turn on five” and it would invoke “TurnOn” command with the GPIO value 5

Here’s an what my screen looked like:

Screen Shot 2017-08-29 at 113849 AMpng

When you are done, click next to go to the next screen

Define the endpoint

This is where you’ll tell Alexa where to ping when it gets one of those commands. It can be any endpoint. To make things easy we’ll use a Lambda function so we only pay when it’s activated as opposed to keeping it running all the time.

Select AWS Lambda ARN, “North American” for region and then we’ll detour to setting up the Lambda function.  We’ll come back to enter in the AWS Lambda ARN. Remember to make note of your Alexa app id as we’ll need it in the Lambda code.

2) Setting up Lambda

AWS Lambda acts as a serverless API. It lets you run code without provisioning servers ahead of time - think of it as an on-demand end point. This is perfect for what we’re doing since we don’t need a full-time server, just something to relay messages from amazon echo to pubnub.

Configuring Lambda

To set it up go to https://console.aws.amazon.com/lambda

Click “Create a Lambda function.” You’ll see a screen asking you to select a blueprint.

Screen Shot 2017-08-04 at 92941 PMpng

Click on “Blank Function.”

Next you’ll be asked to configure the trigger. We want to trigger it from Alexa passing on a voice command and execute code. So for trigger we’ll choose “Alexa Skills Kit”

Screen Shot 2017-08-04 at 93030 PMpng

Now we’ll name our function (eg. “controlgpio”) and select “Node.js 6.10” for the runtime.

Last step is to configure “Lambda function handler and role”, keep it on the defaults:

Screen Shot 2017-08-06 at 44027 PMpng


You should be all set to create your function. Hit “next” and “Create Function.”

At this point you should have an ARN generated that we can use as the “endpoint” for our Alexa Skill. This is where Alexa will ping when the Alexa skill is turned on or hears one of our intent.

To do this go back to Amazon Alexa page and paste the ARN that was just created into the textbox asking for an end point:

Screen Shot 2017-08-29 at 100357 PM copypng

Hit next and you should be all set!

Lambda Code

Go back to Lambda page and provide the code for handling the commands. I’ve made it into a zip to make it easy - 

The zip contains:

  1. Our code for getting a request from Alexa and passing it along to PubNub;

  2. The base Alexa library; and

  3. Required node libraries

After you download the .zip, go to  index.js and modify the app id to the Alexa app id (see above) and PubNub keys (from the first step). You won’t need to change the other files.


Looking at the code you can see that it handles incoming requests (from Alexa), puts them into json and publishes them to a PubNub channel. We will then subscribe to that PubNub channel and listen for any requests. It will be the same for Raspberry Pi or Arduino since it’s just communicating with the PubNub channel.

I’m handling 4 cases:

  • Turning on a particular GPIO (1)

  • Turning off a particular GPIO (2)

  • Turning all on (3)

  • Turning all off (4)

The intents are being translated from words to ints to make it easier for the arduino to handle (mine has very limited memory).

To try out other intents just add them to the GPIO.prototype.intentHandlers array, rezip and reupload. You can see lots of examples in the official Amazon documentation: https://github.com/amzn/alexa-skills-kit-js

When you’re done modifying the app id and PubNub keys, save it, zip all the files - node_modules, index.js and alexa.js (don’t include the containing folder) and then upload the zipped file to Lambda.

You can read more here if you’re interested - https://docs.aws.amazon.com/lambda/latest/dg/nodejs-create-deployment-pkg.html

Testing

Now let's make sure everything is working!

First we’ll verify that there’s nothing horrendously wrong with the lambda code and we can start a new session. To test this select action -> “configure test event”

Screen Shot 2016-11-06 at 65409 PMpng

Paste this into the code section over the existing code (it’s a modified version of the start session template)

Change the applicationId to the Alexa skill you created and hit “save and test”. It should have a green box saying “Execution results: succeeded”.

Now let’s simulate a request from to turn on GPIO 5 (Once again you’ll need to modify the app id to be your app id):


eg.

Screen Shot 2017-08-29 at 93204 PMpng

You should see the success message and if you expand the logs it should say:

{
 "version": "1.0",
 "response": {
   "outputSpeech": {
     "type": "PlainText",
     "text": "Turning on 5"
   },
   "shouldEndSession": true
 },
 "sessionAttributes": {}
}

Yay! We should also make sure it was successfully published to the PubNub channel:

Log back into your PubNub account, select your app and then on the left hand side (at the bottom) click on “debug”. You might already have a client subscribed “hello_world” otherwise just add it and verify that it’s successful by looking for the message being passed to the channel:

Screen Shot 2017-08-29 at 82208 PMpng

You should see your request (1 is the action to “turn on” a particular gpio) published to the channel you’re subscribed to. If you don’t see it make sure you have the right pub/sub keys and you’re communicating with the same channel.

Now let's test all the pieces and make sure the whole flow works by simulating a request from Alexa. In the “service simulator” on the “test” tab in your Amazon Skill pass in the following command:

Turn on 5

To simulate GPIO 5 being switched on:

Screen Shot 2017-08-29 at 115950 AMpng

You should see a response from Lambda and a command in your PubNub debug interface.

Screen Shot 2017-08-29 at 82331 PMpng

Success! Now that we have publishing working lets subscribe to it on the Raspberry Pi and Arduino.

3a) Arduino

Connect to the Internet

First we need to connect Arduino board to the Internet over WiFi. I used an ESP8266 WiFi shield. It doesn’t come with headers soldered to the board, so just buy them seperately and solder them on. Then attach the shield to your Arduino. You could also use an ESP8266 Module and just wire it up directly

Then we’ll use the SparkFunESP8266WiFi library to connect your shield to the internet.

To include the library via the Library Manager, from the drop down select Sketch -> Include Library -> Library Manager and then search for “SparkFunESP8266WiFi” and restart the Arduino IDE. A more thorough guide can be found here: https://www.Arduino.cc/en/Guide/Libraries

Test that your board connects to WiFI with the following code (this is just modified example code from the ESP8266 library):

Upload that code to the Arduino and then open up serial monitor (the little magnifying glass on the top right). You should see it successfully connect to your wifi and then print the results of your get command, in our case an empty array and a timestamp from pubnub:

Screen Shot 2017-08-29 at 120741 PMpng

Setting up your Arduino

Now that we are connected to the internet, let’s set up a visual indicator of whether our code is working. I’m going to use a relay since it has a built in light indicator

The relay has 3 inputs:

  1. Power (+)

  2. Ground (-)

  3. Signal

Connect power to 5v/vcc and ground to one of the GND pins and the GPIO 5 for the signal pin. When GPIO 5 switches to “HIGH” the relay should light up, when it’s on “LOW” it should be dark.

Subscribing to PubNub channel

In order to receive commands from PubNub, you’ll need to subscribe to a channel and wait for Lambda to publish something to the same channel. Arduino has a useful built in “loop” function that predictably loops and executes any code in the function repeatedly, we’ll put our code into it to ping the channel repeatedly and see if anything has been published to it.

You can do this via PubNub’s REST API and do a GET request in the loop or use the PubNub Arduino library and modifying it slightly to make it compatible with ESP8266. In both cases we’ll need to parse the json result.

If you have a mega2560 or board with more than 2k bytes of memory, you can modify the example code provided by PubNub (just change the client to be esp8266) and use the aJson library. My Arduino didn't have enough memory for it so I used the ArduinoJson library. You can see it pings the channel and then if it catches a relevant command it will execute:

You’ll need to configure the code with your wifi and PubNub keys.

Upload to your Arduino and open up serial monitor again, you should see it having successfully connected to your internet and be pinging the channel for any updates.

Now go into PubNub debug console, for the channel select “hello_world” and send a command just like what lambda would give:

Screen Shot 2017-08-13 at 103212 AMpng

You should see the command show up in your serial monitor:

Screen Shot 2017-08-13 at 104903 AMpng

If you don’t see it make sure you’re using the right keys and are on the right channel.

Now let's test the whole thing. As before go to your Alexa testing tab and under "enter utterance" type in:

Turn on 5

Verify that it’s pushed to PubNub as above and that it shows up in lambda serial monitor:

Screen Shot 2017-08-29 at 21533 PMpng

and that the relay turns red!

Success!

3b) Raspberry Pi:

This tutorial is in python, I have one in nodejs coming with a longer tutorial on how to build a tea robot. Hope you find it useful!

Set up

Get a Raspberry Pi, make sure it has a WiFi card (buy a usb one if it doesn’t) and get connected to your WiFi network.

Optional: I’m going to use the same relay as in the arduino tutorial above to indicate whether the pin is on high or low. Connect the power (+) to 5v, ground to gnd and signal wire to pin 5


piwireup2png


Next install the necessary Python libraries on your Raspberry Pi, type the following commands into the command line:

sudo pip install 'pubnub>=4.0.13'

sudo apt-get install python-rpi.gpio

Code

Next, create a Python file called “gpio.py” with the following code:

https://gist.github.com/anonymous/65c82a20ccb5d75d3c8034c8b3e9a438

In terminal run it via “python gpio.py” and you’ll see it say “Waiting for Commands”. Now go into your “service simulator” as above and tell it to turn on pin 5

Screen Shot 2017-08-29 at 90200 PMpng


You should see it show up in your terminal:

2017-08-30-040348_656x416_scrotpng

And see the relay turn on/off.

raspberrypijpg

Success again!

Conclusion

Now that you’ve successfully issued your first voice command to your Raspberry Pi or Arduino, you can quickly iterate on this setup to perform far more elaborate tasks. While I mentioned a few ideas earlier(*cough* robot army *cough*), I’d love to see what creative projects you come up with! If you can, please share videos and comments with me at [email protected]!

views