Using vCloud and Amazon CloudWatch with libcloud

By David Mytton,
CEO & Founder of Server Density.

Published on the 4th March, 2010.

One feature in our server monitoring application, Server Density, and indeed the one I had quite a bit of fun developing, is our cloud provider support. To clarify, we deem any ad hoc VPS, slice, or server supplier a “cloud provider” for the purposes of Server Density.

libcloud

We use a modified version of libcloud to perform API calls to the various cloud server providers. libcloud is an open source, Apache-incubated project, started by the folks over at Cloudkick.

Use within Server Density

There are several places within Server Density where libcloud is used. The most apparent is on the add cloud server screen. On this screen, you’re asked to supply your cloud provider type and corresponding credentials.

Adding a cloud server

When you click the update button, a few things happen. A document is saved to one of our RabbitMQ queues, which is then processed by one of the consumer daemons for that queue. The consumer is written in Python and so fully utilises libcloud. The appropriate cloud provider driver is then created, on which list_nodes is called; the subsequent list of running nodes is stored in a MongoDB collection and populated for the user via a periodic AJAX call.

Two other daemons also use libcloud; the auto-removal and CloudWatch daemons. These both make periodic calls to libcloud drivers either to automatically remove a server from Server Density when it no longer exists, or to pull in metrics from CloudWatch.

Enabling vCloud support

There is a vCloud driver in libcloud, however it is disabled by default. You can enable vCloud support by modifying the providers.py file and adding its driver to the end of the DRIVERS dictionary.

[sourcecode language=”python”]
DRIVERS = {
Provider.DUMMY:
(‘libcloud.drivers.dummy’, ‘DummyNodeDriver’),
Provider.EC2_US_EAST:
(‘libcloud.drivers.ec2’, ‘EC2NodeDriver’),
Provider.EC2_EU_WEST:
(‘libcloud.drivers.ec2’, ‘EC2EUNodeDriver’),
Provider.EC2_US_WEST:
(‘libcloud.drivers.ec2’, ‘EC2USWestNodeDriver’),
Provider.GOGRID:
(‘libcloud.drivers.gogrid’, ‘GoGridNodeDriver’),
Provider.RACKSPACE:
(‘libcloud.drivers.rackspace’, ‘RackspaceNodeDriver’),
Provider.SLICEHOST:
(‘libcloud.drivers.slicehost’, ‘SlicehostNodeDriver’),
Provider.VPSNET:
(‘libcloud.drivers.vpsnet’, ‘VPSNetNodeDriver’),
Provider.LINODE:
(‘libcloud.drivers.linode’, ‘LinodeNodeDriver’),
Provider.RIMUHOSTING:
(‘libcloud.drivers.rimuhosting’, ‘RimuHostingNodeDriver’),
Provider.VOXEL:
(‘libcloud.drivers.voxel’, ‘VoxelNodeDriver’),
Provider.SOFTLAYER:
(‘libcloud.drivers.softlayer’, ‘SoftLayerNodeDriver’),
Provider.VCLOUD:
(‘libcloud.drivers.vcloud’, ‘VCloudNodeDriver’)
}
[/sourcecode]

vCloud works slightly differently than other cloud providers, because it is a VMware product that can be offered as a service by multiple vendors. Because of this, you have to provide a host parameter to your instance of the driver.

[sourcecode language=”python”]
from libcloud.types import Provider
from libcloud.providers import get_driver

vCloud = get_driver(Provider.VCLOUD)
driver = vCloud(‘username’, ‘key’)
driver.connection.host = ‘services.vcloudexpress.terremark.com’
[/sourcecode]

Terremark’s vCloud Express is one such vCloud provider, but you can use any host.

Enabling Amazon CloudWatch

Amazon EC2 is also supported very well by libcloud, however, we needed the ability to pull in CloudWatch metrics whenever possible which is currently not supported by libcloud. Fortunately, since libcloud is open source, we forked the project on Github and added the ability to pull in CloudWatch metrics. You can view the changeset here.

Articles you care about. Delivered.

Help us speak your language. What is your primary tech stack?

Maybe another time