Server monitoring that doesn't suck.

See for yourself

Author Archives: David Mytton

About David Mytton

David Mytton is the founder of Server Density. He has been programming in PHP and Python for over 10 years, regularly speaks about MongoDB (including running the London MongoDB User Group), co-founded the Open Rights Group and can often be found cycling in London or drinking tea in Japan. Follow him on Twitter and Google+.
  1. How to monitor Nginx

    Update: We hosted a live Hangout on Air with Rick Nelson the Technical Solutions architect from NGINX, in which we dug deeper into some of the issues discussed in this blog post. We’ve made the slides and video available, which can be found embedded at the bottom of this blog post.

    Nginx is a popular web server which is often used as a load balancer because of its performance. It is used extensively at Server Density to power our public facing UI and APIs, and also for its support for WebSockets. As such, monitoring Nginx is important because it is often the critical component between your users and your service.

    Monitor Nginx from the command line

    Monitoring Nginx in real time has advantages when you are trying to debug live activity or monitor what traffic is being handled in real time. These methods make use of the Nginx logging to parse and display activity as it happens.

    Enable Nginx access logging

    For monitoring the real time Nginx traffic, you first need to enable access logging by editing your Nginx config file and adding the access_log directive. As a basic example:

    server {
        access_log /var/log/nginx/access_log combined;

    Then restart Nginx and tail the log as requests hit the server to see them in real time:

    tail -f /var/log/nginx/access_log

    Using ngxtop to parse the Nginx access log

    Whilst tailing the access log directly is useful for checking a small number of requests, it quickly becomes unusable if you have a lot of traffic. Instead, you can use a tool like ngxtop to parse the log file for you, displaying useful monitoring stats on the console.

    $ ngxtop
    running for 411 seconds, 64332 records processed: 156.60 req/sec
    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |   64332 |         2775.251 | 61262 |  2994 |    71 |     5 |
    | request_path                             |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    | /abc/xyz/xxxx                            |   20946 |          434.693 | 20935 |     0 |    11 |     0 |
    | /xxxxx.json                              |    5633 |         1483.723 |  5633 |     0 |     0 |     0 |
    | /xxxxx/xxx/xxxxxxxxxxxxx                 |    3629 |         6835.499 |  3626 |     0 |     3 |     0 |
    | /xxxxx/xxx/xxxxxxxx                      |    3627 |        15971.885 |  3623 |     0 |     4 |     0 |
    | /xxxxx/xxx/xxxxxxx                       |    3624 |         7830.236 |  3621 |     0 |     3 |     0 |
    | /static/js/minified/utils.min.js         |    3031 |         1781.155 |  2104 |   927 |     0 |     0 |
    | /static/js/minified/xxxxxxx.min.v1.js    |    2889 |         2210.235 |  2068 |   821 |     0 |     0 |
    | /static/tracking/js/xxxxxxxx.js          |    2594 |         1325.681 |  1927 |   667 |     0 |     0 |
    | /xxxxx/xxx.html                          |    2521 |          573.597 |  2520 |     0 |     1 |     0 |
    | /xxxxx/xxxx.json                         |    1840 |          800.542 |  1839 |     0 |     1 |     0 |

    For more long running monitoring of the logs, Luameter is a better tool that has improved performance for long running monitoring.

    Nginx monitoring and alerting – Nginx stats

    The above tools are useful for monitoring manually but aren’t useful if you want to automatically collect Nginx monitoring statistics and configure alerts on them. Nginx alerting is useful for ensuring your web server availability and performance remains high.

    The basic Nginx monitoring stats are provided by HttpStubStatusModule – metrics include requests per second and number of connections, along with stats for how requests are being handled.

    Server Density supports parsing the output of this module to automatically graph and trigger alerts on the values, so we have a guide to configuring HttpStubStatusModule too. Using this module you can keep an eye on the number of connections to your server, and the requests per second throughput. What values these “should” be will depend on your application and hardware.

    nginx monitoring alerts

    A good way to approach configuring Nginx alerts is to understand what kind of baseline traffic your application experiences and set alerts around this e.g. alert if the stats are significantly higher (indicating a sudden traffic spike) and if the values are suddenly significantly lower (indicating a problem preventing traffic somewhere). You could also benchmark your server to find out at what traffic level things start to slow down and the server becomes too overloaded – this will then act as a good upper limit which you can trigger alerts at too.

    Nginx monitoring and alerting – server stats

    Monitoring Nginx stats like requests per second and number of connections is useful to keep an eye on Nginx itself, but its performance will also be affected by how overloaded the server is. Ideally you will be running Nginx on its own dedicated instance so you don’t need to worry about contention with other applications.

    Web servers are generally limited by CPU and so your hardware spec should offer the web server as many CPUs and/or cores as possible. As you get more traffic then you will likely see the CPU usage increase.

    CPU % usage itself is not necessarily a useful metric to alert on because the values tend to be per CPU or per core. It’s more useful to set up monitoring on average CPU utilisation across all CPUs or cores. Using a tool such as Server Density, you can visualise this and configure alerts so you can be notified when the CPU is overloaded – our guide to understanding these metrics and configuring CPU alerts will help.

    On Linux this average across all CPUs is abstracted out to another system metric called load average. It is a decimal number rather than a percentage and allows you to understand load from the perspective of the operating system i.e. how long processes are waiting for access to the CPU. The recommended threshold for load average therefore depends on how many CPUs and cores you have – our guide to load average will help you understand this further.

    Monitoring Nginx and load balancers with Nginx Plus

    If you purchase a commercial version of Nginx then you get access to more advanced monitoring (and other features) without having to recompile Nginx with the HttpStubStatusModule enabled.

    Nginx Plus includes monitoring stats for connections, requests, load balancer counts, upstream metrics, the status of different load balancer upstreams and a range of other metrics. A live example of what this looks like is provided by Nginx themselves. It also includes a JSON Nginx monitoring API which would be useful for pulling the data out into your own tools.

    Monitoring nginx in real time

    Monitoring the remote status of Nginx

    All of the above metrics monitor the internal status of Nginx and the servers it is running on but it is also important to monitor the experience your users are getting too. This is achieved by using external status and response time tools – you want to know if your Nginx instance is serving traffic from different locations around the world (wherever your customers are) and the kind of response time performance.

    This is easy to do with a service like Server Density because of our in-built website monitoring. You can check the status of your public URLs and other endpoints from custom locations and get alerts when performance drops or there is an outage.

    This is particularly useful when you can build graphs to correlate the Nginx and server metrics with remote response time, especially if you are benchmarking your servers and want to know when a certain load average starts to affect end user performance.

    Monitor Nginx Slides

    Monitor Nginx Video

  2. Sysadmin Sunday 189

    Leave a Comment
  3. How we use Puppet – infrastructure, config, failover, deploys

    Leave a Comment

    We’ve been using Puppet to manage the infrastructure behind Server Density for several years now. It helps us in a number of different ways and although we use it as standard config management, that’s actually only about 25% of our use case.

    We have 4 main uses for Puppet – infrastructure, config management, failover and deploys – each of which I’ll go through here.

    How we use Puppet – Infrastructure

    We first started using Puppet when we moved our environment to Softlayer, where we have a mixture of bare metal servers and public cloud instances, totalling around 75-100. When this was set up, we ordered the servers from Softlayer then manually installed Puppet before applying our manifests to get things configured.

    Although we recently evaluated moving to running our own environment in colo data centres, we have made the decision to switch our environment from Softlayer to Google Cloud. My general view remains that colo is significantly cheaper in the long run but there are some initial capital expenses which we don’t want to spend. We also want to make use of some of the Google products like BigQuery.

    I’ll be writing about this in more detail as we complete the move and am also giving two talks – one in the Bay Area on the 19th June and the other in London on the 2nd of July – about this.

    Using Google Cloud (specifically, Google Compute Engine), or indeed any one of the other major cloud providers, means we can make use of Puppet modules to define the resources within our code. Instead of having to manually order them through the control panels, we can define them in the Puppet manifests alongside the configuration. We’re using the gce_compute module but there are also modules for Amazon and others.

    For example, defining an instance plus a 200GB volume:

    gce_instance { 'mms-app1':
      ensure         => present,
      machine_type   => 'n1-highmem-2',
      zone           => 'us-central1-a',
      network        => 'private',
      tags           => ['mms-app', 'mongodb'],
      image          => 'projects/debian-cloud/global/images/backports-debian-7-wheezy-v20140605',
    gce_disk { 'mms-app1-var-lib-mongodb':
      ensure      => present,
      description => 'mms-app1:/var/lib/mongodb',
      size_gb     => '200',
      zone        => 'us-central1-a',

    The key here is that we can define instances in code, next to the relevant configuration for what’s running on them, then let Puppet deal with creating them.

    How we use Puppet – Config management

    This is the original use case for Puppet – defining everything we have installed on our servers in a single location. It makes it easy to deploy new servers and keep everything consistent.

    It also means any unusual changes, fixes or tweaks are fully version controlled and documented so we don’t lose things over time e.g. we have a range of fixes for MongoDB to work around issues and make optimisations which have been built up over time and through support requests, all of which are documented in Puppet.

    Puppet in Github

    We use the standard module layout as recommended by Puppet Labs, contained within a Github repo and checked with puppet lint before commit so we have a nicely formatted, well structured library describing our setup. Changes go through our usual code review process and get deployed with the Puppet Master picking up the changes and rolling them out.

    Previously, we wrote our own custom modules to describe everything but more recently where possible we use modules from the Puppet Forge. This is because they often support far more options and are more standardised than our own custom modules. For example, the MongoDB module allows us to install the server and client, set options and even configure replica sets.

      include site::mongodb_org
      class {'::mongodb::server':
        ensure    => present,
        bind_ip   => '',
        replset   => 'mms-app',
      mount {'/var/lib/mongodb':
        ensure  => mounted,
        atboot  => true,
        device  => '/dev/sdb',
        fstype  => 'ext4',
        options => 'defaults,noatime',
        require => Class['::mongodb::server'],
      mongodb_replset { 'mms-app':
        ensure  => present,
        members => ['mms-app1:27017', 'mms-app2:27017', 'mms-app3:27017']

    We pin specific versions of packages to ensure the same version always gets installed and we can control upgrades. This is particularly important to avoid sudden upgrades of critical packages, like databases!

    The Server Density monitoring agent is also available as a Puppet Forge module to automatically install the agent, register it and even define your alerts.

    All combined, this means we have our MongoDB backups running on Google Compute Engine, deployed using Puppet and monitored with Server Density.


    How we use Puppet – Failover

    We use Nginx as a load balancer and use Puppet variables to list the members of the proxy pool. This is deployed using a Puppet Forge nginx module we contributed some improvements to.

    When we need to remove nodes from the load balancer rotation, we can do this using the Puppet web UI as a manual process, or by using the console rake API. The UI makes it easy to apply the changes so a human can do it with minimal chance of error. The API allows us to automate failover in particular conditions, such as if one of the nodes fails.

    How we use Puppet – Deploys

    This is a more unusual way of using Puppet but has allowed us to concentrate on building a small portion of the deployment mechanism, taking advantage of the Puppet agent which runs on all our servers already. It saves us having to use custom SSH commands or writing our own agent, and allows us to customise the deploy workflow to suit our requirements.

    It works like this:

    1. Code is committed in Github into master (usually through merging a pull request, which is how we do our code reviews).
    2. A new build is triggered by Buildbot which runs our tests, then creates the build artefacts – the stripped down code that is actually copied to the production servers.
    3. Someone presses the deploy button in our internal control panel, choosing which servers to deploy to (branches can also be deployed) and the internal version number is updated to reflect what should be deployed).
    4. /opt/puppet/bin/mco puppetd runonce -I is triggered on the selected hosts and the Puppet run notices that the deployed version is different from the requested version.

    5. The new build is copied onto the servers.


    Status messages are posted into Hipchat throughout the process and any one of our engineers can deploy code at any time, although we have a general rule not to deploy non-critical changes after 5pm weekdays and after 3pm on Fridays.

    There are some disadvantages to using Puppet for this. Firstly, the Puppet agent can be quite slow on low spec hardware. Our remote monitoring nodes around the world are generally low power nodes so the agent runs very slowly. It’s also eventually consistent because deploys won’t necessarily happen at the same time, so you need to account for that in new code you deploy.

    Puppet is most of your documentation

    These four use cases mean that a lot of how our infrastructure is set up and used is contained within text files. This has several advantages:

    • It’s version controlled – everyone can see changes and they are part of our normal review process.
    • Everyone can see it – if you want to know how something works, you can read through the manifests and understand more, quickly.
    • Everything is consistent – it’s a single source of truth, one place where everything is defined.

    It’s not all of our docs, but certainly makes up a large proportion because it’s actually being used, live. And everyone knows how much they hate keeping the docs up to date!

    Puppet is the source of truth

    It knows all our servers, where they are, their health status, how they are configured and what is installed.

  4. How to keep on top of security updates


    Every computer user needs to stay on top of updates for their apps, browsers, and OSs. As a consumer, it’s easy to do this – the best browsers like Chrome auto update in the background and over the air updates on iOS make sure most people get updates quickly, and easily. These kind of mass market updates tend to be well tested so you can update without fear of bricking.

    The life of a sysadmin is a little more complex though. Although the mainstream OSs provide mature package management features, there are often a lot of frequent updates across the OS, kernel updates which require reboots, application (e.g. databases) updates which need testing and library release which need recompiling or updating codebases.

    I’ve written before about our canary concept for rolling out updates because we deploy them manually rather than letting the OS auto update whenever it likes, but how do you discover new releases in the first place?

    OS X Update

    Critical security update notifications

    The first step is to separate general OS/library/app updates with critical security releases. Like the recent Heartbleed bug, these need to be deployed as quickly as possible.

    The first point of call is the OS security announcements. We use Ubuntu and so have the choice of subscribing to the Ubuntu Security Announcements list, or via RSS. In these announcements I look out for keywords like “remote” or “denial of service” because these mean that there’s an external risk, contrasted to an exploit which requires a local user. This is more difficult to exploit because it first requires access to our servers, which is restricted to our ops team.

    Debian has a similar mailing list as do other Linux distros like Red Hat. Windows has an update mechanism built in which makes it easier than with multiple Linux distros.

    Ubuntu updates

    Updating software packages

    Most of the software products and libraries we use are installed through system packages using apt. This is our preferred method of installing because it makes it easy to manage via Puppet, is standardised across all our servers, offers signing of packages and importantly, gives us a single place to apply upgrades. This is one of the reasons why our monitoring agent recommended installation method is via our OS packages.

    A lot of packages are provided as part of the OS release from Ubuntu package maintainers, so they get rolled into the security notifications.

    The disadvantage of this approach is that you often don’t get the very latest version through the OS vendor packages. To work around this, vendors will usually provide their own official repositories (e.g. our MongoDB installations are through the MongoDB, Inc repos).

    apt-get upgrade

    Everything else is manual

    A few things are installed manually, such as PHP pecl packages for some of our legacy apps. We can update them using pecl but have to keep an eye on the release mailing lists for those specific packages.

    And where we run “on-premise” versions of products we also subscribe to the relevant mailing lists. Our policy is to prefer SaaS where possible but some things aren’t available like this – we run our own Puppet Enterprise server (with an announce mailing list) and have the MongoDB Backup Service running within our infrastructure.

    We also keep an eye on pages like the MongoDB Alerts and mailing list.


    What about general security mailing lists?

    You could subscribe to mailing lists like BUGTRAQ or the CVE lists but these are really high volume and aren’t specific enough to our environment. We can get what we need from the vendors, mostly from Ubuntu.

    There are also commercial products like the Ubuntu Landscape product which we used for a while but it was too expensive to maintain a support contract for so many servers, with limited value. So long as we can stay up to date with the most important security releases, we can be sure that we have a secure infrastructure as regards to software updates.

  5. What’s new in Server Density – Apr 2014

    Leave a Comment

    Each month we’ll round up all the feature changes and improvements we have made to our server and website monitoring product, Server Density.

    Elastic dashboard graphs

    Static dashboard graphs are good if your cluster never changes but if you have an elastic environment where new instances appear and disappear, you can use the new elastic graph widget which will add and remove items as they appear and disappear in your account.

    You can specify a search term for your inventory of devices and service checks, which can be a regular expression with wildcards. Once you create your graph, it will be automatically updated based on this search term so if new servers appear that match, they will automatically be added to the graph. This is useful if you set a prefix for your cluster and always name new servers based on that prefix. Read more.


    Android app

    Following our iPhone app, last week we released our Android app to allow you to manage alerts and get push notifications directly to your Android devices. Find out more here.

    Android Server Monitoring App Use Case

    HipChat notifications

    We use HipChat pretty extensively at Server Density and we’ve now released native integration so you can have your alerts posted directly into your HipChat rooms. Read how to set this up.


    Slack notifications

    In addition to HipChat, you can also have notifications sent directly to Slack. Read how to set this up.

    Ansible playbooks

    Several customers have released their Ansible playbooks to allow you to install the agent and manage your monitoring using Ansible.

    Server Density 日本



    (We have launched a Japanese language version of this blog!)

  6. Network performance at AWS, Google, Rackspace and Softlayer

    Comments Off

    This post was originally published on GigaOm.

    Compute and storage are essentially commodity services, which means that for cloud providers to compete, they have to show real differentiation. This is often achieved with supporting services like Amazon’s DynamoDB and Route 53, or Google’s BigQuery and Prediction API, which complement the core infrastructure offerings.

    Performance is also often singled out as a differentiator. Often one of the things that bites production usage, especially in inherently shared cloud environments, is the so-called “noisy neighbor” problem. This can be other guests stealing CPU time, increased network traffic and, particularly problematic for databases, i/o wait.

    In this post I’m going to focus on networking performance. This is very important for any serious application because it affects the ability to communicate and replicate data across instances, zones and regions. Responsive applications and disaster recovery, areas where up-to-date database replication is critical, require good, consistent performance.

    It’s been suggested that Google has a massive advantage when it comes to networking, due to all the dark fibre it has purchased. Amazon has some enhanced networking options that take advantage of special instance types with OS customizations, and Rackspace’s new Performance instance types also boast up to 10 Gbps networking. So let’s test this.


    I spun up the listed instances to test the networking performance between them. This was done using the iperf tool on Linux. One server acts as the client and the other as the server:

    Server: iperf -f m -s
    Client: iperf -f m -c hostname

    The OS was Ubuntu 12.04 (with all latest updates and kernel), except on Google Compute Engine, where it’s not available. There, I used the Debian Backports image.

    The client was run for three tests for each type – within zone, between zones and between regions – with the mean average taken as the value reported.

    Amazon networking performance

    Amazon networking performance

    Amazon’s larger instances, such as the c3.8xlarge tested here, support the enhanced 10 GB networking, however you must use the Amazon Linux AMI (or manually install the drivers) within a VPC. Because of the additional complexity of setting up a VPC, which isn’t necessary on any other provider, I didn’t test this, although it is now the default for new accounts. Even without that enhancement, the performance is very good, nearing the advertised 10 Gbits/sec.

    However, the consistency of the performance wasn’t so good. The speeds changed quite dramatically across the three test runs for all instance types, much more than with any other provider.

    You can use internal IPs within the same zone (free of charge) and across zones (incurs inter-zone transfer fees), but across regions, you have to go over the public internet using the public IPs, which incurs further networking charges.

    Google Compute Engine networking performance

    Google networking performance

    Google doesn’t currently offer an Ubuntu image, so instead I used its backports-debian-7-wheezy-v20140318 image. For the f1-micro instance, I got very inconsistent iperf results for all zone tests. For example, within the same us-central-1a zone, the first run showed 991 Mbits/sec, but the next two showed 855 Mbits/sec and 232 Mbits/sec. Across regions between the US and Europe, the results were much more consistent, as were all the tests for the higher spec n1-highmem-8 server. This suggests the variability was because of the very low spec, shared CPU f1-micro instance type.

    I tested more zones here than on other providers because on April 2, Google announced a new networking infrastructure in us-central-1b and europe-west-1a which would later roll out to other zones. There was about a 1.3x improvement in throughput using this new networking and users should also see lower latency and CPU overhead, which are not tested here.

    Although 16 CPU instances are available, they’re only offered in limited preview with no SLA, so I tested on the fastest generally available instance type. Since networking is often CPU bound, there may be better performance available when Google releases its other instance types.

    Google allows you to use internal IPs globally — within zone, across zone and across regions (i.e., using internal, private transit instead of across the internet). This makes it much easier to deploy across zones and regions, and indeed Google’s Cloud platform was the easiest and quickest to work with in terms of the control panel, speed of spinning up new instances and being able to log in and run the tests in the fastest time.

    Rackspace networking performance

    Rackspace networking performance

    Rackspace does not offer the same kind of zone/region deployments as Amazon or Google so I wasn’t able to run any between-zone tests. Instead I picked the next closest data center. Rackspace offers an optional enhanced virtualization platform called PVHVM. This offers better i/o and networking performance and is available on all instance types, which is what I used for these tests.

    Similar to Amazon, you can use internal IPs within the same location at no extra cost but across regions you need to use the public IPs, which incur data charges.

    When trying to launch x2 120 GB Performance 2 servers at Rackspace, I hit our account quota (with no other servers on the account) and had to open a support ticket to request a quota increase, which took them about an hour and a half to approve. For some reason, launching servers in the London region also requires a separate account, and logging in and out of multiple control panels soon became annoying.

    Softlayer networking performance

    Softlayer networking performance

    Softlayer only allows you to deploy into multiple data centers at one location: Dallas. All other regions have a single facility. Softlayer also caps out at 1 Gbps on its public cloud instances, although its bare metal servers do have the option of dual 1 Gbps bonded network cards, allowing up to 2 Gbps. You choose the port speed when ordering or when upgrading an existing server. They also list 10Gbit/s networking as available for some bare metal servers.

    Similarly to Google, Softlayer’s maximum instance size is 16 cores, but it also offers private CPU options which give you a dedicated core versus sharing the cores with other users. This allows up to eight private cores, for a higher price.

    The biggest advantage Softlayer has over every other provider is completely free, private networking between all regions whereas all other provider charge for transfer out of zone. When you have VLAN spanning enabled, you can use the private network across regions, which gives you an entirely private network for your whole account. This makes it very easy to deploy redundant servers across regions and is something we use extensively for replicating MongoDB at Server Density, moving approx 500 Mbits/sec of internal traffic across the US between Softlayer’s Washington and San Jose data centers. Not having to worry about charges is a luxury only available with Softlayer.

    Who is fastest?

    Who is fastest?

    Amazon’s high spec c3.8xlarge really gives the best performance across all tests, particularly within the same zone and region. It was able to push close to the advertised 10 GB throughput, but the high variability of results may indicate some inconsistency in the real-world performance.

    Yet for very low cost, Google’s low spec f1-micro instance type offers excellent networking performance: ten times faster than the terrible performance from the low spec Rackspace server.

    Softlayer and Rackspace were generally bad performers overall, but at least Rackspace gets some good inter-zone and inter-region performance and performed well for its higher instance spec. Softlayer is the loser overall here with low performance plus no network-optimized instance types. Only their bare metal servers have the ability to upgrade to 10 Gbits/sec network interfaces.

    Mbits/s per CPU?

    CPU allocation is also important. Rackspace and Amazon both offer 32 core instances, and we see good performance on those higher spec VMs as a result. Amazon was fastest for its highest spec machine type with Rackspace coming second. The different providers have different instance types, and so it’s difficult to do a direct comparison on the raw throughput figures.

    An alternative ranking method is to calculate how much throughput you get per CPU. We’ll use the high spec inter-zone figures and do a simple division of the throughput by the number of CPUs:

    Mbits per CPU

    The best might not be the best value

    If you have no price concerns, then Amazon is clearly the fastest, but it’s not necessarily the best value for money. Google gets better Mbits/sec per CPU performance, and since you pay for more CPUs, it’s a better value. Google also offers the best performance on its lowest spec instance type, but it is quite variable due to the shared CPU. Rackspace was particularly poor when it came to inter-region transfer, and Softlayer isn’t helped by its lack of any kind of network-optimized instance types.

    Throughput isn’t the end of the story though. I didn’t look at latency or CPU overhead and these will have an impact on the real world performance. It’s no good getting great throughput if it requires 100 percent of your CPU time!

    Google and Softlayer both have an advantage when it comes to operational simplicity because their VLAN spanning-like features mean you have a single private network across zones and regions. You can utilize their private networking anywhere.

    Finally, pricing is important, and an oft-forgotten cost are the network transfer fees. This is free within zones for all providers, but only Softlayer has no fees for data transfer across zones and even across regions. This is a big saver.

  7. Android Server Monitoring App released

    Comments Off

    After the release of our monitoring app for iPhone 2 months ago, the same app is now available on Android!

    It allows you to see your notification center on the move so you can quickly see open and closed alerts assigned to you and across your whole account. It includes push notification alerts and custom sounds so you can always be woken up!

    Android Server Monitoring App Use Case

    The app is free for all v2 customers (and users on the trial). Find out more on the product page of our website.

  8. How we use Hipchat to work remotely


    Server Density started as a remote working company, and it wasn’t until our 3rd year that we opened an office in London. Even now, most of the team still work remotely from around the UK, Portugal and Spain so we use a variety of tools to help us stay connected and work efficiently together.

    One of the most important is Hipchat. We use this as a chat room but it’s also our company news feed. Everything that goes on during the day (and night) gets posted and we use a variety of rooms for different purposes.

    The main room

    Everyone is always logged in here during their working day and it also acts as a real time news feed of what is going on. General chat happens here but it’s mostly a way for everyone to see what’s happening. This is particularly useful if you go away and come back later because you can see what has happened.

    Main Hipchat room

    We use various integrations and the Hipchat API to post in events from things like:

    • Github activity: commits, pull requests, comments.
    • Buildbot: Build activity.
    • Deploys: From our own custom build system, we can see when deploys are triggered (and by whom) and then when they get deployed to each server.
    • Signups: New trial signups get posted in as we often like to see who is using the product.
    • Account changes: New purchases and package changes
    • JIRA: All our issue tracking and development work is tracked with JIRA, which posts some activity like new issues and status changes.
    • Zendesk: New support tickets and Twitter mentions so everyone can keep an eye on emerging issues.
    • Alerts: We use some of our competitors to help us maintain an independent monitor of our uptime, and pipe in alerts using the new Server Density HipChat integration and PagerDuty.

    Ops war room

    All incidents from our alerting systems get piped into a war room so we can see what’s going on in real time, chronological order and not be distracted by other events. The key here is maintaining a sterile cockpit rule so we use this room to only discuss ongoing incidents. This also is useful for other people (e.g. support) to track what’s happening without interrupting the responders.

    Server Density hipchat


    We have a bot that runs in all of our chat rooms. It’s fairly simple and based off Hubot but allows us to do things like query Google Images or check the status of Github.


    At a glance

    We use our own server monitoring ops dashboard on the TV in the office using a Chromecast and I also have an iPad at my desk which constantly shows the Hipchat room, so I can see things as they happen throughout the day.

    Server Density office

  9. Dealing with OpenSSL bug CVE-2014-0160 (Heartbleed)


    Yesterday a serious vulnerability in the OpenSSL library was disclosed in CVE-2014-0160, also dubbed the Heartbleed Vulnerability. Essentially this means you probably need to regenerate the private keys used to create your SSL certificates, and have them reissued by your certificate authority.


    This isn’t a difficult task but does take some time to get OpenSSL updated across all your servers, then go through the process to generate, reissue and install certificates across all locations they are deployed.

    We have completed this process for all of our websites and applications and for Server Density v2 we use perfect forward security which should protect against retrospective decryption of previous communications.

    However, in the latest release of our server monitoring iPhone app we enabled certificate pinning which means until our latest update is approved by Apple, the app will not log in. You will still receive push notifications for alerts but attempts to log in to the app will fail. Certificate pinning embeds our SSL certificate within the app which is then used to prevent man in the middle attacks – the certificate that is returned through the API calls to our service is verified against the known certificate embedded in the app.

    We discussed the best way to approach the reissue of certificates this morning and considered holding off to allow us to submit a new build to Apple with pinning disabled, then do a future update with the new certificate. However, we felt that the security vulnerability was severe enough that we should patch it for all our users at the expense of causing a small number of users to be unable to use the iPhone app for a few days.

    We have requested Apple expedite the review process but it still takes at least 24 hours to get a new release out. In the meantime, you should check to see if your OpenSSL version is vulnerable and if so, update!

  10. MongoDB on Google Compute Engine – tips and benchmarks


    Over the last 4 years running MongoDB in production at Server Density, I’ve been able to work on deployments on dedicated hardware, VMs and across multiple cloud providers.

    The best environment has always been dedicated servers because of the problems with host contention, particularly with CPU and disk i/o but Google has been quite vocal about the consistency and performance of their Compute Engine product, particularly about how they’ve eliminated the noisy neighbour problem with intelligent throttling.

    So I thought I’d try it out to see how MongoDB performs on Google Compute Engine.

    Google Compute Engine Logo

    Testing the Write Concern – performance vs durability

    The MongoDB Write Concern is historically controversial because Mongo was originally designed to get very high write throughput at the expense of durability but this wasn’t well documented. The default was changed a while back and it now gives you an acknowledgement that a write was accepted, but is still quite flexible to allow you to tune whether you want speed or durability.

    I am going to test a range of write concern options to allow us to see what kind of response times we can expect:

    Unacknowledged: w = 0 AND j = 0

    This is a fire and forget write where we don’t know if the write was successful and detection of things like network errors is uncertain.

    Acknowledged: w = 1 AND j = 0 (the default)

    This will give us an acknowledgment that the write was successfully received but no indication that it was actually written. This picks up most errors e.g. parse errors, network errors etc.

    Journaled: w = 1 AND j = 1

    This will cause the write to wait until it has been both acknowledged and written to the journal of the primary replica set member. This gives you single server durability but doesn’t guarantee the data has been replicated to other members of your cluster. In theory you could have a data center failure and lose the write.

    Replica acknowledged: w = 2 AND j = 0

    The test will give us an idea how long it takes for the write to be acknowledged on the replica set primary and acknowledged by at least 1 other member of the replica set. This gives us some durability across 2 servers but in theory the write could still fail on both because we are not doing a check for the write hitting the journal.

    Replica acknowledged and Journaled: w = 2 AND j = 1

    This ensures that the write is successfully written to the primary and has been acknowledged by at least one of the replica set members.

    Replica acknowledged with majority: w = majority AND j = 0

    In a multi datacenter environment you want to know that your writes are safely replicated. Using the majority keyword will allow you to be sure that the write has been acknowledged on the majority of your replica set members. If you have the set deployed evenly across data centers then you know that your data is safely in multiple locations.

    Replica acknowledged with majority and journaled: w = majority AND j = 1

    Perhaps the most paranoid mode, we will know that the write was successfully acknowledged by the primary and was replicated to a majority of the nodes.

    Environment configuration

    Replica sets

    Real world applications use replica sets to give them redundancy and failover capabilities across multiple data centers. To accurately test this, the test environment will involve 4 data nodes across 2 zones: x2 in the us-central1-a zone and x2 in the us-central1-b zone.

    In a real deployment you must have a majority to maintain the set in the event of a failure, so we should deploy a 5th node as an arbiter in another data center. I’ve not done this here for simplicity.

    Google Compute Engine

    I tested with the n1-standard-2 (2 vCPUs and 7.5GB RAM) and n1-highmem-8 (8 vCPUs and 52GB RAM) instance types – with the backports-debian-7-wheezy-v20140318 OS image.

    Be aware that the number of CPU cores your instance has also affects the i/o performance. For maximum performance then you need to use the 4 or 8 core instance types even if you don’t need all the memory they provide.

    There is also a bug in the GCE Debian images where the default locale isn’t set. This prevents MongoDB from starting properly from the Debian packages. The workaround is to set a default:

    sudo locale-gen en_US.UTF-8
    sudo dpkg-reconfigure locales

    Google Persistent Disks

    It’s really important to understand the performance characteristics of Google Persistent Disks and how IOPs scale linearly with volume size. Here are the key things to note:

    • At the very least you need to mount your MongoDB dbpath on a separate persistent disk. This is because the default root volume attached to every Compute Engine instance is very small and will therefore have poor performance. It does allow bursting for the OS but this isn’t sufficient for MongoDB which will typically have sustained usage requirements.
    • Use directoryperdb to give each of your databases their own persistent disk volume. This allows you to optimise both performance and cost because you can resize the volumes as your data requirements grow and/or to gain the performance benefits of more IOPs.
    • Putting the journal on a separate volume is possible even without directoryperdb because it is always in its own directory. Even if you don’t put your databases on separate volumes, it is worth separating the journal onto its own persistent disk because the performance improvements are significant – up to x3 (see results below).
    • You may be used to only needing a small volume for the journal because it uses just a few GB of space. However, allocating a small persistent disk volume will mean you get poor performance because the available IOPs increase with volume size. Choose a volume of at least 200GB for the journal.
    • If you split all your databases (or even just the journal) onto different volumes then you will lose the ability to use snapshots for backups. This is because the snapshot across multiple volumes won’t necessarily happen at the same time and will therefore be inconsistent. Instead you will need to shut down the mongod (or fsync lock it) and then take the snapshot across all disks.

    I’ve run the testing several times with different disk configurations so I can see the different performance characteristics:

    1. With no extra disks i.e. dbpath on the default 10GB system volume
    2. With a dedicated 200GB persistent disk for the dbpath
    3. With a dedicated 200GB persistent disk for the dbpath and another dedicated 200GB persistent disk for the journal

    Test methodology

    I wrote a short Python script to insert a static document into a collection. This was executed 1000 times and repeated 3 times. The Python timeit library was used to complete the tests so the fastest time was taken, as per the docs indicating that the mean/standard deviation of the 3 test cycles is not that useful.

    Results – MongoDB performance on Google Compute Engine


    Test results - n1-standard-2


    Test results - results-n1-highmem-8


    There’s not much difference in performance when you’re using a single persistent disk volume until you start increasing the durability options because acknowledged and unacknowledged writes are just going to memory. When you increase the write concern options then you become limited by the disk performance and it’s clear that splitting the journal onto a separate volume makes a significant difference.

    The increased performance of the larger n1-highmem-8 instance type with 8 cores vs 2 cores of the n1-standard-2 is also apparent in the figures – although the actual difference is quite small, it is still a difference which would likely help in a real world environment.

    The key takeaway is that performance decreases as durability increases – this is a tradeoff. For maximum performance with durability then you can take advantage of Compute Engine’s higher spec instance types and larger, separate persistent disk volumes per database / journal.