Written by pessoa

This post is based on our talk at Puppet Camp Ghent. See our separate page for all our talk videos and slides about how Puppet is used to manage the Server Density monitoring infrastructure.

We’ve been using Puppet to manage our infrastructure of around 100 servers for over a year now. This originally started with manually building manifests to match the setup in our old hosting environment at Terremark so we could migrate to Softlayer. Over the last few months we’ve been refactoring it to make use of Forge modules and combining roles on some servers to reduce the total number. Using Forge modules allows us to reuse community code rather than reimplementing everything ourselves.

Puppet Forge

While preparing our infrastructure for Server Density v2, out in the next few months, one of the areas that required special consideration was our load balancer. We currently use Pound, but the new product required additional functionality that we could only get from Nginx.

Nginx is an event driven web server with inbuilt proxying and load balancing features. It has inbuilt native support for FCGI and uWSGI protocols allowing us to run Python WSGI apps in fast application servers such as uWSGI.

Unlike Pound and Varnish, new builds of Nginx have support for both WebSockets and Google’s SPDY standard plus supported third party modules for full TCP/IP socket handling, all of which allows us to pick and mix between our asynchronous WebSocket Python apps and our upstream Node.js proxy.

This obviously had to be both fully deployable and fully configurable through Puppet.

Searching for a Puppet solution

The choice was between writing our manifest – yet another one to the collection – or refraining from reinventing the wheel by reaching out to the community and looking whether our problem had already been solved (or at least some sort of kick start we could stand on).

Since we believe more in reuse than roll our own, a visit to the Forge was inevitable.

Our setup

• We run Puppet Enterprise using our own manifests. These are pulled from our Gihub repo by the puppet master.
• Puppet Console and Live Management get used quite intensively to trigger transient changes such as failover switches (video from PuppetConf 2012).

Integrating it all

Having selected the Puppetlabs nginx module, we needed to add it to our Puppet Enterprise (PE) setup. We need to A) get the actual code in and B) be able to have it run on existing nodes.

There are 2 ways to do A):

1. puppet module install puppetlabs/nginx
2. git submodule add https://github.com/puppetlabs/puppetlabs-nginx.git on the existing puppet master modules folder

This decision is eased by looking at the module and realising that we will be making enhancements to it. Since we’ll fork puppetlabs-nginx for those changes, the best way to get that fork into the puppet master is by having a git submodule.

Now for B). We found out quickly that PE Console does not yet support parameterised classes such as this one. So we’re left with the option of doing a merge between our site.pp (which is empty) and the console, it being an ENC and all. But that would kill our ability to use the Console to trigger transient changes as mentioned before. We want to continue to have our node parameters managed on the Console and not on site.pp. For completeness, node inheritance could also be used but would probably get quite messy.

The solution we settled on is having a class hierarchy:

class serverdensity-nginx
{
    class { 'nginx': }
    nginx::resource::upstream {
       'socky_rack':
          ensure => present,
          members => $lbTargetHosts,
    }

...

Our own serverdensity-nginx is used by PE Console and it then includes the nginx class, allowing us to use both the Console node parameters and the nginx module functionality.

Trigger transient changes using the Console

One of the best aspects of this solution is that it allows to overcome nginx lack of a control interface, keeping the operations functionality we were used to with Pound.

Nginx requires the configuration files to change and it then loads the configuration on reload. To, eg. remove one node from the load balancer rotation, one would need to edit the corresponding configuration file and trigger a nginx reload.

With Puppet, this is achieved changing the node, or group, parameter using the Console and then triggering the node to run. Puppet will then reload nginx. And best, the in flight connection are not terminated by a reload.

Alternatives, or what could we have done differently?

Later we learned that Puppet Labs Nginx module is derived from James Fryman’s module. It might have been better to pick that one up since the Puppet Labs module hasn’t been updated since the 30th June 2011. Apparently Puppet Labs wants to ensure that all modules in the Puppet Labs namespace on the Forge are best-in-class and actively maintained. They mentioned that they hope to kick off this project in early 2013.

We have made some modifications to the module:

• Several SSL improvements. Thank you Qualys SSL Labs!
• Additional custom logging
• Some bug corrections and beautification

And you can find our fork on Github.

Enjoy this post? You may also like Designing and printing (dot) notebooks