Browser Controlled Singing, Animatronic Jack O'Lanterns and Light Show With Arduino and Raspberry Pi

The jack o'lanterns themselves sing by opening and closing their mouths using servos. Their eyes are also animated. Lights, such as strings of mini-lights, can be made to flash to the music as well.

The lights connect to a solid state relay (SSR) control box. The control wires for the SSRs, the eyes, and the mouths all connect to an Arduino. The Arduino accepts a live serial stream that is passed out to each of the controls. That stream comes from some sort of computer. In my case, I'm using a Raspberry Pi running Falcon Pi Player (FPP) to run an animation sequence and to play the associated music. Amplified speakers must be connected to the Pi to play the music. FPP runs with a web interface, and you will control your playlists from your phone or computer wirelessly.

Finally, the animation sequence is created by specialized animation software that can export the 'fseq' files FPP expects. I'm using Vixen, but you can also use XLights. Both are free, and it's a matter of taste. The animation must be done frame by frame (though there are tools like Papagayo that can help alleviate the tedium). I made mine frame by frame, channel by channel. Be prepared to spend some time doing this.

3 More Images

How you do this really depends on 1) the look you want, and 2) the kind of pumpkins you got. I highly recommend buying hollow foam pumpkins. You can use the solid foam pumpkins, too, but you'll need to carve out a space for your servos and a route for your lights. You won't be able to get away with real pumpkins. They're too wet and too heavy.

1. Draw a draft of your pumpkin's face in pencil.
2. With a utility knife or a small saw, cut the pumpkin in half along the mouth. You will want a cut that makes the jack o'lantern's mouth open naturally, so cut along any teeth, etc.
3. Attach hinges to the back of the cut to make a working jaw. Some trial and error is probably required here, but you'll want them as close to the middle of the back as possible. I needed two hinges on the larger jack o'lantern and one on the smaller jack o'lantern.
4. Drill out holes for the eye LEDs. Optionally, you can cut out the eyes completely and use something to diffuse the LEDs for a better look. In version 1 of my jack o'lanterns, I simply used a 1/4 inch drill to make holes into which I glued bare 5mm LEDs.
5. Make a discrete hole in the rear bottom of the pumpkin through which the wires will be routed.
6. Get artistic, and decorate your jack o'lantern! I let my kids decorate two of mine.
7. Repeat for each jack o'lantern.

3 More Images

1. Using wood or foamboard scraps, make a platform for your servo inside the pumpkin at the middle of the bottom. The servo position is again a matter of trial and error. You want the platform high enough that your servo horns and the arm we will fashion has space to move up and down. You want it forward as much as possible, because the longer the control arm gets (the radial part), the less weight it can lift. However, the shorter the control arm, the smaller the opening in the mouth will be while singing.
2. Fashion a control arm that will work with your servo. I used wire hangers from dry cleaning, because these attach to the servo control horns well. I needed to use a small drill bit to open the control horn holes a little. I used a "D" shape to hold the stiff wire to the horns. This has the advantage of being fairly secure, yet quickly removable. It also attaches to the servo horn in two places, adding some leverage and security.
   
   The control arm extends away from the servo enough to lower speed and increase the mouth opening width.
   
   The upgoing portion of the arm ends in a circle so that the inside top of the jack o'lantern rides on it smoothly. There is no need to attach the top of the pumpkin to the arm.
3. Hot glue the servo to the platform. If you want to repurpose the servo after Halloween, it should pry away pretty easily, and the hot glue will be fairly easy to remove from the servo case.
4. At this point, if you have some easy way to activate the servo, it's probably a good idea to test it. Make sure the max opening is to your liking.
5. Repeat for each jack o'lantern.
6. Wait until you are sure the servo is zeroed before screwing the servo horn to the servo output shaft with the included screw.

6 More Images

This part is slightly trickier than you might think because we are powering the LEDs with the Arduino directly. Each output pin should source no more than 40 mA at 5V. For red and yellow LEDs, all you need is a simple series circuit, because their forward voltage is around 2V at 20 mA. For any other colors (e.g. the Halloween colors) you are looking at 3.4V or so. You'll need to connect the LEDs in parallel in order to power them with 5V. At 20 mA each you can still get away with powering a 2 LED parallel circuit directly from the Arduino. Note that these are typical values. Your LEDs may be different.

1. Wire 2 LEDs in series with a resistor. You can use Ohm's law to figure out the resistor. You need to drop (5V - 2V - 2V) = 1V for two red LEDs at 20 mA, so R = 1V/0.02mA = 50 Ohms. This is the resistor you need for max brightness. I used 220 Ohm resistors because I had them sitting on my bench, and the eyes are more than bright enough. Don't want to blind your audience, after all. Give enough slack between the LEDs to fit the eyes of your jack o'lantern.
   
   Ensure that the cathode (-) of the first LED connects to the anode (+) of the other. Resistor direction doesn't matter.
2. Alternatively, if using green, blue, or white LEDs, wire them in parallel. Each LED needs its own resistor. I used 220 Ohm resistors here again, and it was fine. You can use smaller resistors for brighter eyes, but 220 Ohms was sufficient for me. In detail, solder the anode of one resistor to the incoming + power, the cathode to a 220 Ohm resistor, and the other end of the resistor to a short length of wire to ground. For the second LED, the circuit is the same, but move the short length of wire to the anode side of the second LED.
3. Test your circuit using the 5V and GND connections on your Arduino.
4. If not using a diffuser, press the LEDs into the eye holes you created earlier and hot glue them into place.

Diffuser (Optional)

For my largest pumpkin, I wanted the eyes to stand out more. Single points of light work fine for the little pumpkins, but they're kind of lost on the big one. For the diffusers I used these parts:

• Lenses from a free pair of sunglasses
• cotton balls (give a neat swirly ghost-eye effect)
• black card stock

Assembly:

1. Cut out holes for the eyes that are smaller than the sunglasses lenses
2. Hot glue the lenses over the holes from the inside
3. Stretch the cotton balls until they cover the whole lens, and hot glue into place (using as little glue as possible)
4. Place the LEDs with the sides facing the lenses, rather than the bright ends. Glue the LEDs into place using the portions of the wires that extend beyond the lenses.
5. Cut pieces of black card stock large enough to cover the whole eye assembly, and glue over the whole setup. This card stock keeps light out of the pumpkin interior. Lighting up the interior sounds cool, but everyone can see the wires and the servo assembly, spoiling the effect.

2 More Images

You need to run 4 wires to each jack o'lantern

1. Power for servo
2. Signal for servo
3. Power/signal for eyes
4. Ground (shared by both)

The LEDs can be powered directly from the Arduino.

I used old cat-3 phone lines to wire my jack o'lanterns. You will need enough wire to reach from your jack o'lantern to the control box. Keep in mind you probably want to conceal the controls for effect, and the control box will need to be in range of your wifi. Multiply this length by the number of jack o'lanterns you have.

1. Strip back the casing to reveal the individual conductors.
2. Solder LED power directly to one of the strands in the Cat 3 cable.
3. Solder a 3 pin male header to three of the remaining lines. We are making a plug for the servo connector, so the order of the connections should match the order of your servo's connections. The order varies by servo manufacturer, so consult your documentation. I'm using two different brands of servo, so in my photos, you may notice that the header wire order changes.
4. Solder the ground wire from the eyes to the ground pin on your header, too.
5. Optionally, coat the soldered connections with hot glue for insulation and strength.
6. At the other end, solder a 1 pin male header onto each conductor.
7. Wrap a small piece of masking tape around each conductor and label its function. Trust me, this step will save you many headaches.

Notes:

• You need to trim your wires such that there is slack in the lines going to the eyes. If you don't, the wires will prevent the mouth from opening.

I built my light show using homemade solid state relays (SSR). You can of course use plain old relays, but they make a lot of noise. That's important for this project because the relay box is going to be right next to your jack o'lanterns.

I've already explained this part of the build in another one of my Instructables here.

As I said there, BE CAREFUL. This part of the project involves household mains, and it can KILL you if you are careless. Don't touch anything inside the SSR box when it's plugged in. Many triacs, including the ones I used here, have metal tabs that are live when the triac is active.

After you've built and tested your SSR, you need to build a wiring harness. A multi-conductor cord like a phone or ethernet cable works well here again. You'll need a conductor for each light you want to control separately and one for ground. This cord does NOT carry mains current. It is safe to touch assuming you've wired things correctly. I soldered single pin male headers to the far end of this harness. As with the jack o'lanterns, this harness must be long enough to reach the control box.

Finally, plug your lights into the SSR box outlets. These can be strands of Halloween lights, or lighted props. Actually, they can be anything that plugs in, so long as you don't overload the SSR box.

This Arduino sketch includes bits and pieces found around the web. Part of it comes from this other singing pumpkin Instructable, and part of it comes from zparticle's more well known sketch. And there are some of my own adjustments and modifications on top of all of those sketches.

Get the sketch from GitHub here.

I've documented the code with lots of comments, but in summary, this is what's going on:

1. We load the servo library and the watchdog timer library.
2. Define our channels and mark how many are currently active. I've defined handy names for each pin. This is especially useful for the Arduino Uno, where the pins aren't exactly in numerical order with regards to how we'd like to use time. I've also added the ability to define a lot more channels (things we'll control) than we actually use. This is great for future improvements, and for testing. You can just set the number of channels N you need, and then drop them into the first N slots of the 'channels' array.
3. Define whether we're testing or not. This is just a convenience that tells the code to treat channels we mark as servos as non-servos. Creating the actual animation is time consuming, and it helps to set up a test breadboard where the servos are represented by simple LEDs. Setting the testing variable lets those LEDs light up normally (see the step on animation later).
4. Define a servo mask. This is an array that maps servo numbers to channel numbers, or marks channels that are not servos. The mask array should be at least as long as your channel array.
5. Define an array of servos.
6. Define the max and min opening angles for each servo. This will be different for each pumpkin due to differences in size and in how you built your control rod. You'll have to set this through trial and error.
7. Define a marker for the beginning of an animation frame. This can be anything you like. I'm following zparticle here and using a '~!' to mark the beginning. You'll need to tell your animation software to prefix each frame with this. Note that this is not at all necessary (the other pumpkin tutorial above doesn't use it). It just a cheap error checking that helps keep the pumpkins in sync with the animation stream.
8. Setup our serial connection. Make note of the baud rate you choose here, because you'll need to match your animation stream baud rate with it.
9. Setup a watchdog timer to reboot the Arduino if it freezes for more than one second. Again, not necessary, but it's just some cheap insurance. Especially if you're leaving the pumpkins unattended.
10. In the loop, we watch the serial port for incoming animation frames, and then pass a byte to each channel. If the channel is not a servo, but is PWM, it just gets the byte. If it's not a servo and not PWM, we do a cheap conversion to on or off and then send it out. If it is a servo, we convert the byte to an angle and use the servo library to send it out.

That's it. The stream representing the animation and the audio are both handled by the Pi.

3 More Images

NOTE: This step is OPTIONAL. You can skip ahead to the Vixen animation section and run the entire show directly from Vixen. FPP eliminates the need for a computer, gives you controls on your phone, and does scheduling and playlists. It also makes the entire setup cheaper and smaller.

Installing Falcon Pi Player (FPP)

If you're familiar at all with the Raspberry Pi, you'll know it's a full computer. FPP is an entire linux distribution that you place on your Pi's SD card, just like you would install Raspbian OS.

Follow the installation instructions here: http://falconchristmas.com/forum/index.php?board=8...

FPP will need to connect to your home network, either by ethernet or wifi. Wifi will obviously be more portable. The instructions are at the same link.

Configuring FPP

Everything is controlled through a web interface. If everything has been installed correctly, you should be able to get to the control website running from your Pi by opening a browser on your computer or phone to http://fpp

Setting up the output for our Arduino sketch

FPP knows how to control a lot of different off the shelf light show devices. It also knows how to generate a generic serial stream right out of the box, which is perfect. That's exactly what our Arduino sketch is waiting for. We just need to configure the serial stream so that it matches what the Arduino expects.

1. Go to Input/Output Setup -> Channel Outputs
2. Click the "Other" tab
3. Click the Add button
4. Select "Generic Serial" from the Output Type dropdown menu
5. Set Start Channel to 1
6. Set Channel Count to 8 (or whatever your channel count is)
7. Set the port to the USB port your Arduino will connect to. This may take trial and error, because the names don't obviously map to real-world USB ports.
8. Set the Speed to 115200 (unless you chose a different baud rate in your Arduino sketch)
9. Enter '~!' (without the quotes) in the Header box. This is the header we indicated in our Arduino sketch.
10. Tick the 'active' checkbox

Uploading animation and music

1. Go to Content Setup -> File Manager
2. Under the "Sequences" tab, click "Select Files" to upload the fseq files produced by Vixen or xLights.
3. Under the "Audio" tab, click "Select Files" to upload the audio files synced to your sequences.

Setting up a playlist

1. Go to Content Setup -> Playlists
2. Enter a new playlist name
3. Add entries one at a time by selecting the right combination of sequences and audio.

Starting and stopping the show

1. Go to the main page (Status/Control -> Status Page)
2. Select a playlist from the list of those available
3. Click the "Play" button
4. For more control, you can use the Scheduler under Content Setup -> Scheduler to automate when the show starts and stops.

Going Further

This doesn't even scratch the surface of what FPP is capable of. Consult the falcon website for more help, or just click the help button on the FPP control page. Everything is documented reasonably well.

I'm using Vixen 3 here, but xLights is also a viable option. I'm not giving a full tutorial on Vixen, because there are already great tutorials around the web.

If you've ever used Premiere, Final Cut Pro, iMovie, Audacity, FLStudio, or Flash, you probably already know 90% of what to do here. Vixen gives you a timeline that you must add channels to, and in that channel you define frames by assigning a value between 0-255, which will become the distance the jack o'lantern's mouth opens, or the brightness of a PWM light. Synchronize this with the audio, and you've got an animation sequence.

I'm going to warn you ahead of time: this part is tedious. There are a number of sites where people have posted their own sequences, but most of them are for driving large, multichannel light shows, and not animated servos. There is good news, though. Songs are repetitive. Cut and paste is your friend. Even parts of the songs with different words can be cut and pasted if the rhythm is the same, because we're only opening and closing a mouth. We don't have to worry about lip-syncing.

I've created a video tutorial embedded above.

Build a test proxy for your show on a breadboard

Just use an LED to represent each channel. Connect your breadboard ground to the Arduino ground, and plug the cathode of each LED into the ground rail on the breadboard. Plug the anode into a row, add a resistor across the gutter (220 or 330 ohm will work), and the connect a jumper wire from the each resistor to the channels you defined in your sketch. Set testing mode on your Arduino sketch, and you're good to go. You'll have to set up Vixen to send a serial stream to the Arduino, just like we did with FPP.

This is far easier than dealing with the actual servos.

Streaming from Vixen to the Arduino

You can stream directly from Vixen to the Arduino. You do not need the Raspberry Pi or FPP. In fact, you will need to do this for animation, unless you trust the built-in animation display in Vixen.

The central control box contains the Arduino and the power bus for your servos. It can, given room, also contain your Raspberry Pi, but keep in mind the Pi will run warm.

I used a sandwich meat container flipped upside down. The opaque top becomes a floor to attach things to, and the translucent shell becomes a domed cover that you can see test LEDs through.

1. Using the screw holes provided on the Arduino, attach it to the underside of the container lid as shown. I used enameled wire as twist ties.
2. Attach your power bus bar to underside of the lid using wire or hot glue.
3. Wire the bus ground to one of the Arduino GND pins.
4. Solder power and ground wires to your barrel hack, and attach the ends to the + and - sides of the power bus.
5. Label the + and - strips.
6. Make holes in the (former) bottom of the container for the Arduino's USB cable and for the control lines from the jack o'lanterns and the light show.
7. Route control wires through the holes.
8. Attach all control ground wires to the - strip on the power bus.
9. Attach servo power lines to the + strip of the bus
10. Attach all control pins to the appropriate pins on the Arduino. If you are using the sketch provided here unaltered, then attach as follows:
    1. Pin 9: Pumpkin 1 mouth
    2. Pin 10: Pumpkin 2 mouth
    3. Pin 11: Pumpkin 3 mouth
    4. Pin 3: Pumpkin 1 eyes
    5. Pin 5: Pumpkin 2 eyes
    6. Pin 6: Pumpkin 3 eyes
    7. Pin 12: Light set 1
    8. Pin 13: Light set 2

If you're curious about the odd pin order, we need PWM (analog, essentially) output to control the mouths and eyes, and the Arduino has 6 PWM outputs. The light sets are digital, so they can go in any of the remaining pins.

1. Attach jack o'lanterns to a base using hot glue or screws. I used an old shelf to keep things portable
2. Attach Arduino to control box (if desired). I used enameled wire from my toolkit to twist-tie the Arduino to the top of a reusable container, and flipped it upside-down. In fact, I've kept my Arduino this way for years. It's a good, aesthetically pleasing, protective box for it.
3. Attach your power bus to your control box. I hot-glued my homemade bus.
4. Wire the barrel jack to your bus and mark the ground and positive sides.
5. Connect the bus ground to your Arduino ground.
6. Make a hole for the Arduino serial cable and the control wires for the pumpkins and the SSR box.
7. Push the control wire header pins into the correct pin on the Arduino. These were the pin numbers you set in your Arduino sketch.
8. Connect the servo power and ground to the correct sides of your power bus.
9. Connect a serial line from the Arduino to the Raspberry Pi. Be sure to select the USB port you specified in the FPP setup. This will also provide the power for your Arduino.
10. Connect a set of amplified speakers (computer speakers work well) to the Pi's audio jack.
11. Plug everything in. You should have 1) power for the Raspberry Pi, 2) Power for servos 3) Power for the SSR box, and 4) Power for your speakers.
12. Log into FPP and start your playlist!

FPP supports a lot of different options. It can control many commercial controllers. Even more useful, it supports GPIO based events. With a little creative use of FPP, you can have your jack o'lanterns react to the environment. For instance, you might have them react to a button press or a motion sensor. All it takes is some setup in the web interface and some shell scripting at worst. FPP takes care of handling the GPIO and attaching the event to a script.

Although I've shown an Arduino UNO in all my shots, in this year's version, I'm switching to an Arduino Mega 2560. This gives me a lot more I/O pins, and therefore the ability to control a lot of additional lights and motors. There are 14 PWM pins, which gives me 8 additional analog controls, one of which will control a blacklight prop I've described here. I'm also adding a hardware, real world skip button, and a button which will cause the pumpkins to stop whatever they're doing and scream. This is supported by FPP via something called "effect sequences", which are just plain sequences that are designed to play outside of the normal loop. Finally, the ultimate goal in this year's version is to have my other props broadcast via WiFi (using an ESP8266) when they are triggered, and have the pumpkins play the same scream sequence as if the prop scared them, too.

If you'd like to give a little control to your guests, you can create a hardwired remote. Or, if you prefer, you could use this schematic as a stepping stone for controlling the show with any external control. This works because FPP supports GPIO, and supports it well. Any GPIO can be connected to any "event", which is FPP parlance for running either an effect sequence or any shell script. Not only that, FPP comes with a number of scripts in a repository that you can install for your project. Many of these call the command line version of the fpp program, and they're usually just a few easily modified lines.

I created a remote with 3 buttons: Rewind, Fast-Forward, and one that is labeled "Push to see the kids jump", which makes the jack o'lanterns scream loudly.

For this remote, I used the following parts:

• A mini Altoids can
• 3x momentary buttons
• 3x 10k ohm resistors
• 1x 1k ohm resistor
• Some cat-5e cable
• 1x 1 pin female header
• 1x 4 pin female header
• Hot glue

We're triggering low, so the 3 10k resistors are pull-up resistors. The 1k is a current limiting resistor. To build:

1. Cut a piece of perfboard to size so that it fits inside the tin with room on the edges for the lid to close
2. Attach the three buttons to the perfboard by bending the button pins to grasp the board.
3. Mark the lid of the tin for holes that match your button placement. I spaced my 3rd button away from the first two.
4. Drill 3 holes in your altoids tin. I placed mine to one side because I wanted to leave room for labels.
5. Drill a hole in one end of the tin for the cable.
6. Paint your tin if desired. Now's the best time.
7. Push the cable through the hole you created for it.
8. Solder the buttons according to the schematic.
9. Hot glue the perfboard and cable in place, taking care not to interfere with the button motion.
10. Close the tin.
11. If desired, create labels and glue in place. I used white glue on the back, and thinned white glue over the top of the labels to "decoupage" them in place and protect them from dirty fingers.
12. On the other end, solder the single pin header to a length of wire, and solder the other end to one side of all three pull-up 10k resistors
13. Solder the other ends of the resistors to three of the adjacent pins on the 4 pin header.
14. Solder the 3 button wires of the remote wires from the cable to the header wires to which you soldered the pull up resistors.
15. Solder the ground wire to one end of the 1k resistor, and solder the other end to the remaining pin.
16. Take note of which pin goes to which button.
17. Plug the single header into pin 1 on your Raspberry Pi. This must go to 3.3V, not 5V.
18. Plug the 4 pin header into the adjacent ground pin and pins 22, 27, and 17. In terms of board pin numbers, these are pins 5, 7, 9, and 11. If that's confusing, it's because the GPIO number is not the same as the physical pin numbers.

That's it for the physical part of the remote. Now we need to set up the software side.

1. Start your Pi and navigate a browser to FPP
2. Go to Content Setup->Script Repository Browser
3. Install "GoToNextPlaylistItem.sh", "GoToPreviousPlaylistItem.sh". You may also want to grab "CheckIfPlaying.sh". Be sure to look at each of these. The first two are one line. They just call "fpp -c next" and "fpp -c prev" respectively.
4. Go to Input/Output Setup->GPIO Inputs
5. If you've followed my setup, you should activate BCM 17, 27, and 22. FPP will tell you the daemon needs to restart. Let it.
6. Now you need to attach some kind of action to each GPIO event. You can choose to trigger when the GPIO voltage goes high (rising) or low (falling). We have set ours up to trigger low, so from the falling pulldown box, pick the appropriate script from the ones you installed. We used BCM 17 for next, so choose GoToNextPlaylistItem.sh for that one. Choose the "previous" script for BCM 27. We haven't created a script for the scream button yet.
7. Start a playlist and try out your buttons. My kids hated that the Next button didn't start the show if it was at a complete stop, so I combined "CheckIfPlaying.sh" with GoToNextPlaylistItem.sh so that FPP only sends "fpp -c next" if it's playing. If not, it will start my big playlist using "fpp -P All", where "All" is the name of my playlist.
8. Finally, you need to create a scream playlist. This can be as simple as two commands. You can just copy GoToNextPlaylistItem.sh and change the contents to "fpp -c stop" and "fpp -P Scream" where "Scream" is the name of your screaming playlist. Again, that wasn't good enough for me. I wanted to pumpkins to start again after the scream, preferably as close to where they stopped as possible. The closest I got so far was to check if anything was playing, record the playlist name and the playing entry number, and then start it up again after the screams. The playing song will start from the beginning again, but that's close enough. My script looks like this:

#/bin/sh
SLEEP_TIME=3
###########################################################

STATUS=$(fpp -s)

if [ "x${STATUS}" != "xfalse" ]
then
	# We received a response from fppd so parse it
	PLAYLIST=$(fpp -s | cut -f4 -d,)
	ITEM=$(fpp -s | cut -f8 -d,)
	ITEM=$(($ITEM-1))

	# do not get caught in a loop
	if [ "x${PLAYLIST}" != "xScreams" ]
	then
		fpp -c stop
		fpp -P Screams
		sleep $SLEEP_TIME
		echo "sleep $SLEEP_TIME"

		if [ "x${PLAYLIST}" != "xNo playlist scheduled." ]
		then
			# We were playing a playlist before
			fpp -P $PLAYLIST $ITEM
			echo "fpp -P $PLAYLIST $ITEM"
		fi
	fi
fi