Sending async SMS with Node.js and cloud devops bots

By David Mytton,
CEO & Founder of Server Density.

Published on the 6th September, 2012.


Tom wasn’t the only one having fun at LeedsHack last month! One idea idea I’ve had a for while now, mostly since seeing a talk on Github’s Hubot at a conference last year and writing hippybot , is the concept of a SaaS platform that acts as a hosted devops bot.

First, let me explain what I mean by “devops bot”; Hubot has exemplified this concept, especially with GitHub using it to deploy their entire infrastructure and as a major tool in their ops belt. I see such bots as two way interfaces between human devs, ops and everyone else, and processes which both help to analyse your infrastructure and drive it. It doesn’t matter what communication protocol/platform they work over (IRC, HipChat, Campfire, email etc) but it helps if they work with the comms you and your team actually use every day.

Sure you could build a web tool that lets you deploy the main app to production, or gives you a reading of any slow DB queries, or even tells you if the office coffee machine empty; but just think for a moment the power of doing this in the chat room your team is already discussing the latest site slowness or wondering if there’s coffee to be had. Not only do you have a very personable interface in the form of a bot (a fake person performing as a member of the team) but your team have direct access to it too and everyone sees the output of common commands (disregarding commands which may require extra permissions or return data over private message to the executor).

So where does a hosted service come in? Surely even with bots like Hubot you can just configure and launch on a cloud platform? GitHub themselves run on Heroku – and there’s even a webapp by Jonathan Hoytthat does the pre-config for you. However, the same benefits of SaaS should apply to this concept as much as any other, and in fact I think it applies extremely well.

So, my idea for LeedsHack was to produce an alpha version of a service with the following features:

  • Allows you to spin up a cloud hosted bot from a web UI.
  • Allows you to configure all aspects of the bot, not just deployment configuration like which port it will run on Heroku.
  • Allows you to relaunch with configuration changes.
  • More importantly allows you make some configuration changes, and without restarting, have the changes applied live.

The service would later provide custom plugins/addons/integrations but that wouldn’t be required for this proof of concept.

As an added bonus I also wanted to provide SMS integration for the bot, but I admit this was purely because event sponsor MediaBurst had 5 Google Nexus 7s on offer as prizes for the best hack(s) using this SMS sending/receiving web service!

Although I have issues with Hubot’s limitations and some of its internals, on the day it supplied the most well rounded and feature complete base bot, with a large ecosystem of addons, multiple connection type adapters and in a stack I had experience with (Node.js and CoffeeScript).

Before working the problems of automating and extending a Hubot deployment, I decided another component should be a way of easily accessing MediaBurst’s API from Node.js. As they didn’t already have a JavaScript library I’d need to write one. This was the first time I’d written a Node.js library, mostly working with CoffeeScript before, I knew the majority of well accepted libs are written in pure JS and any self respecting lib is expected to be fully async using callbacks and any appropriate async methods, taking advantage of Node’s call stack.

Node.js comes with excellent HTTP request handling built in, but the request library by Mikeal Rogers wraps all the usual configuration and handling for a HTTP requests into some handy shortcut functions, so that was a no-brainer for me. The difficulty came in how to best generate and parse the XML payloads for requests and responses to/from the API.

There are a wide variety of XML utils (DOM, SAX, OO etc.) for Node.js, but all I needed was something fast and easy to understand. Generating the correct payload for API calls turned out to be fairly simple, especially when I settled on using xmlbuilder.js in a recursive serialize function that would take a JS object/hash, looping over its attributes and depending on the attribute type create a different leaf node or call itself to generate another tree, e.g. to generate the following payload:

[sourcecode language=”xml”]
<?xml version="1.0" encoding="UTF-8"?>
<Content>Hello World</Content>

We just call:

[sourcecode language=”javascript”]serialize({
‘Message’: {
‘Key’: ‘key’,
‘SMS’: {
‘To’: ‘441234567890’,
‘Content’: ‘Hello World’

However, parsing proved difficult as the format being returned wasn’t entirely strict XML, and sometimes included unprintable characters, which a fair few of the parsing libs (even the ones using the excellent and flexibile libxml and expat libraries) couldn’t handle. But after a few horrible yak-shaving hours spent staring at angular brackets I settled on using the lovely xml2js lib, trading some speed and parsing flexibility for ease of access to parsed data. That is as long as you use the following to disable strict mode in the underlying sax parser:

[sourcecode language=”javascript”]var parser = new xml2js.Parser();
parser.saxParser.strict = false;[/sourcecode]

Other than these niggles, the library is a fairly standard base object with methods that take callback closures, fire off some requests and then passes back data (or an error in the event) to the supplied callback. Sending multiple SMS to one or more numbers is accomplished in the API itself by serializing an array of numbers for the To part of the payload, so I didn’t see a reason to turn this part of the handler into an async process, leaving that bit up to MediaBurst.

For the result check out node-clockwork, also available on npm.

Back to my main idea, I ended up creating a Vagrant package, which consisted of an Ubuntu Lucid basebox, a Hubot git checkout and a crude bash script to install some apt dependencies, build hubot and then run it with my HipChat config info. This allowed me to test deploying Hubot without running the lengthy build process via a Heroku dynamo for long periods at a time – I could just keep the already built files between reruns ofvagrant up.

My customisation was a simple HTTP listener defining an endpoint for set config variables inside Hubot’s “redis brain”, along with an SMS plugin employing my clockwork lib to send messages from hubot within a HipChat chatroom. My main app itself was a simple Node.js + app that on click of one button would send a command to vagrant to start up a pre-configured VM with hubot set to start, and then wait for a longpoll HTTP connection to Hubot. Via this connection it would send a JSON payload containing config information (and config updates).

As the next 24 hours tolled on, the caffeine got consumed (in lieu of real sleep) and my wits got dulled, my feature list was reduced to these steps:

  • Spin up instance (via heroku), Hubot connects to HipChat
  • Send JSON payload, for now restricted to list of plugins and basic plugin config
  • Subsequent submits of the form with plugin data in would send the payload again and reload the plugins data
  • Hubot announces this on HipChat
  • Hubot responds to “sms” command which triggers my SMS plugin*, which sends message to registered users…
  • …if a user replies to a message or sends a new one to a special phone number provided by MediaBurst, a webhook would be fired by MB, which in this case is a URL on my Node/ app which then sends a slightly different payload for message updates to the Hubot instance.
  • The endpoint for this call on Hubot then reports the message to a registered user on HipChat.

* Why make this a plugin? I had planned to make the SMS plugin a full adaptor, letting Hubot “connect and communicate via SMS” just like any other transport, however Hubot currently doesn’t support multiple adaptors, so my workaround was to make it a plugin

The UI for all of this was boiled down to a single page with a textarea containing JSON for a plugin list, and a button to fire the data off. This was the app I presented, along with node-clockwork, on the Sunday. Image provided by my artist friend Joff Oliver:

Zom – the smart zombie for your cloud. If you’re interested I literally asked Joff to draw me a zombie professor, and that he must have a pipe, an hour later I get the image attached to an email…now that’s service!

The Github repo for Zom doesn’t contain the code I wrote for all this, as I decided it wasn’t particularly public-ready and that I’d like to work on the idea more in the future, but rest assured I’ll hopefully get a chance to work on this again and hope to release all the source (even if I make it into a service too). My code for dynamically configuring the plugin list in Hubot from an array stored in redis and read during the plugin load method is definitely a candidate for cleaning up and submitting a pull request.

All in all I had a sleep-deprived blast at LeedsHack this year, and as it turned out I did in fact win a Nexus 7 from the folks at MediaBurst for creating node-clockwork (although only found out the following day as I’d departed for home early before the awards)!

Articles you care about. Delivered.

Help us speak your language. What is your primary tech stack?

Maybe another time