Server monitoring that doesn't suck.

See for yourself

Author Archives: David Mytton

About David Mytton

David Mytton is the founder of Server Density. He has been programming in PHP and Python for over 10 years, regularly speaks about MongoDB (including running the London MongoDB User Group), co-founded the Open Rights Group and can often be found cycling in London or drinking tea in Japan. Follow him on Twitter and Google+.
  1. How to monitor Apache

    Leave a Comment

    Apache is perhaps the most well known and widely deployed web servers, having originally been released back in 1995 and currently deployed on a large number of web servers (although losing ground to NGINX). As an important part of the classic LAMP stack, it is a critical component in your web serving architecture, and if you’re not currently – you should be monitoring Apache.

    Enabling Apache monitoring with mod_status

    Most of the tools for monitoring Apache require the use of the mod_status module. This is included by default but needs to be enabled. You will need to specify an endpoint in your Apache config:

    
    
    <Location /server-status>
    
      SetHandler server-status
      Order Deny,Allow
      Deny from all
      Allow from 127.0.0.1
    
    </Location>
    
    

    This will make the status page available at http://localhost/server-status on your server. We have a full guide to configuring this. Be sure to enable the ExtendedStatus directive to get full access to all the stats.

    Monitoring Apache from the command line

    Once you have enabled the status page and verified it is working above, you can make use of command line tools to monitor the traffic on your server in real time. This is useful for debugging issues and examining traffic as it happens.

    The apache-top tool is a popular method of achieving this. It is often available as a system package e.g. apt-get install apachetop but can also be downloaded from the source, as it is only a simple Python script.

    Apache monitoring and alerting – Apache stats

    Using apache-top is useful for real time debugging and examining what is happening on your server right now, but it is less useful if you want to collect statistics over a period of time. This is where a monitoring product such as Server Density will come in. Our monitoring agent supports parsing the Apache server status output and can give you statistics on requests per second and idle/busy workers.

    Apache has several process models but the most common is to have worker processes running idle waiting to service requests. As more requests come in then more workers will be launched to handle them, up to a configured maximum. At that point the requests will be queued and your visitors will experience delays. This means it’s important not just to monitor the raw requests per second but also how many idle workers you have.

    A good way to approach configuring Apache alerts is to understand what kind of baseline traffic your application experiences and set alerts around this e.g. alert if the stats are significantly higher (indicating a sudden traffic spike) and if the values are suddenly significantly lower (indicating a problem preventing traffic somewhere). You could also benchmark your server to find out at what traffic level things start to slow down and the server becomes too overloaded – this will then act as a good upper limit which you can trigger alerts at too.

    Apache monitoring and alerting – server stats

    Monitoring Apache stats like requests per second and worker status is useful to keep an eye on Apache itself, but its performance will also be affected by how overloaded the server is. Ideally you will be running Apache on its own dedicated instance so you don’t need to worry about contention with other applications.

    Web servers are generally limited by CPU and so your hardware spec should offer the web server as many CPUs and/or cores as possible. As you get more traffic then you will likely see the CPU usage increase, especially as Apache workers take up more CPU time and are distributed across the available CPUs and cores.

    CPU % usage itself is not necessarily a useful metric to alert on because the values tend to be per CPU or per core and you may have many cores. It’s more useful to set up monitoring on average CPU utilisation across all CPUs or cores. Using a tool such as Server Density, you can visualise this and configure alerts so you can be notified when the CPU is overloaded – our guide to understanding these metrics and configuring CPU alerts will help.

    On Linux this average across all CPUs is abstracted out to another system metric called load average. It is a decimal number rather than a percentage and allows you to understand load from the perspective of the operating system i.e. how long processes are waiting for access to the CPU. The recommended threshold for load average therefore depends on how many CPUs and cores you have – our guide to load average will help you understand this further.

    Monitoring the remote status of Apache

    All of the above metrics monitor the internal status of Apache and the servers it is running on but it is also important to monitor the experience your users are getting too. This is achieved by using external status and response time tools – you want to know if your Apache instance is serving traffic from different locations around the world (wherever your customers are) and the kind of response time performance. You will then know at what stage you need to add more capacity, either by increasing the capabilities of the Apache server or by adding more into a load balanced cluster.

    This is easy to do with a service like Server Density because of our in-built website monitoring. You can check the status of your public URLs and other endpoints from custom locations and get alerts when performance drops or there is an outage.

    This is particularly useful when you can build graphs to correlate the Apache and server metrics with remote response time, especially if you are benchmarking your servers and want to know when a certain load average starts to affect end user performance.

  2. What’s new in Server Density – Summer 2014

    Leave a Comment

    We’ve been a bit quiet over the last few months but have still been working on improvements and new functionality to our server and website monitoring product, Server Density. This post summarises what we added over the summer and what’s coming up soon.

    Log search beta

    One of the first things you do when responding to an alert or tracking down performance problems is look at the server logs. Current log management products are expensive and complex to set up, so we’re pleased to announce the beta of our log search functionality.

    Log search uses the existing Server Density agent to tail your logs and make them searchable from within your account. There’s a new dedicated search view so you can search by device, or you can view the logs from individual device views. Later, logs will automatically be displayed as part of a new, upcoming alert incident view.

    If you’re interested in trying this out, please fill out this short form to get into the beta.

    Server Density log search

    Google Cloud integration

    We released our integration into Google Cloud and Google Compute Engine which allows you to manage your instances and get alerts on instance and disk state changes. You can also sign up for $500 in free Google Compute Engine credits.

    Google Cloud graphs

    Snapshots

    Click on any data point on your device graphs and then click the Snapshot link, and it will take you through to a view of what was happening on that server at that exact point in time. You can also click the Snapshot tab to go to the latest snapshot and then navigate backwards and forward through each time point.

    Server snapshot

    Linux agent 1.13.4

    A number of fixes have been released as part of the latest Linux agent release, including better handling of plugin exceptions and more standards compliance for init scripts. MongoDB over SSL is also now supported. See the release notes.

    Chef cookbook improvements

    There are a range of improvements to the official Chef cookbook which include better support for EC2 and Google auto scaling and support for managing plugins through Chef. This is available on the Chef Supermarket and has had almost 100,000 downloads in the last 10 days.

    Puppet module improvements

    The official Puppet module has also had improvements to make it work better with Google Cloud. It is also available on the Puppet Forge.

    App performance improvements

    A lot of work has been done behind the scenes to improve the performance of the product generally. This ranges from optimising requests and connections in the UI, upgrades to the hardware powering the service to moving all our assets onto a CDN. We have a few more improvements still to release but this all goes towards our goal of having response times as close to instantaneous as possible.

    Onboarding and help popups

    We retired our old app tour with new in-app popup bubbles to help you learn more about functionality. Blank slates have been redesigned and we have more improvements to help show off some of the great functionality coming soon.

    How to monitor xyz

    We’re running a series of free webinars through Google Hangouts to cover how to monitor a range of different technologies. We started with MongoDB but our next upcoming hangout will be on how to monitor Nginx. Many more hangouts will be scheduled over the next few months and you can watch them back through our Youtube channel.

    Redesigned multi factor authentication setup

    The flow for setting up a new multi factor token has been redesigned to make it clearer how to proceed through. We highly recommend enabling this for extra security – passwords are no longer enough!

    Enable MFA

    Improved cloud actions menu

    Actions taken within Server Density are separated from actions taken on the Cloud Provider level to ensure commands aren’t sent accidentally.

    Cloud actions

    Delete confirmations

    Previously it was too easy to take the delete actions which could lead to accidentally deleting a device. We’ve improved the confirmation requirements for this.

    delete

    Auto refreshing graphs

    All graphs, on the device overview and on the dashboard, now auto refresh so you can keep the window open and see the data show up immediately.

    What’s coming next?

    We’ll be returning to our monthly post schedule for “What’s new” as we start releasing some of the things we’ve been working on over the last few months. This includes permissions and a range of new alerting functionality, starting with tag based alerts and group recipients. Lots of interesting new functionality to be announced before the end of the year!

  3. Automated Google Cloud and Google Compute Engine monitoring

    Leave a Comment

    Today we’re releasing the Server Density integration into Google Compute Engine as an official Google Cloud Platform Technology Partner. Server Density works across all environments and platforms and is now fully integrated into Google’s cloud infrastructure products, including Compute Engine and Persistent Disks, to offer alerting, historical metrics and devops dashboards to Google customers.

    Google Cloud graphs

    Server Density customers can connect their Google Cloud accounts to automatically monitor and manage instances across Google data centers alongside existing environments and other cloud providers. Many customers will run systems across multiple providers in a hybrid setup, so Server Density is uniquely placed to help with that because even though we have specialist integration into Google, it works well anywhere – cloud, hybrid and on-prem.

    $500 credit for Google/Server Density customers

    Server Density normally starts at $10/m to monitor Linux, Windows, FreeBSD and Mac servers but Google Cloud customers can monitor up to 5 servers for free for life (worth over $500/year). Google is also offering Server Density customers $500 in credits to trial Google Cloud Platform. To find out more and sign up, head over to our website for details.

  4. How to monitor MongoDB

    Leave a Comment
    Update: We hosted a live Hangout on Air with Paul Done from MongoDB discussing how to monitor MongoDB. We’ve made the slides and video available, which can be found embedded at the bottom of this blog post.

    We use MongoDB to power many different components of our server monitoring product, Server Density. This ranges from basic user profiles all the way to high throughput processing of over 30TB/month of time series data.

    All this means we keep a very close eye on how our MongoDB clusters are performing, with detailed monitoring of all aspects of the systems. This post will go into detail about the key metrics and how to monitor your MongoDB servers.

    MongoDB Server Density Dashboard

    Key MongoDB monitoring metrics

    There is a huge range of different things you should keep track of with your MongoDB clusters, but only a few that are critical. These are the monitoring metrics we have on our critical list:

    Oplog replication lag

    The replication built into MongoDB through replica sets has worked very well in our experience. However, by default writes only need to be accepted by the primary member and replicate down to other secondaries asynchronously i.e. MongoDB is eventually consistent by default. This means there is usually a short window where data might not be replicated should the primary fail.

    This is a known property, so for critical data, you can adjust the write concern to return only when data has reached a certain number of secondaries. For other writes, you need to know when secondaries start to fall behind because this can indicate problems such as network issues or insufficient hardware capacity.

    MongoDB write concern

    Replica secondaries can sometimes fall behind if you are moving a large number of chunks in a sharded cluster. As such, we only alert if the replicas fall behind for more than a certain period of time e.g. if they recover within 30min then we don’t alert.

    Replica state

    In normal operation, one member of the replica set will be primary and all the other members will be secondaries. This rarely changes and if there is a member election, we want to know why. Usually this happens within seconds and the condition resolves itself but we want to investigate the cause right away because there could have been a hardware or network failure.

    Flapping between states should not be a normal working condition and should only happen deliberately e.g. for maintenance or during a valid incident e.g. hardware failure.

    Lock % and disk i/o % utilization

    As of MongoDB 2.6, locking is on a database level, with work ongoing for document level locking in MongoDB 2.8. Writes take a global database lock so if this situation happens too often then you will start seeing performance problems as other operations (including reads) get backed up in the queue.

    We’ve seen high effective lock % be a symptom of other issues within the database e.g. poorly configured indexes, no indexes, disk hardware failures and bad schema design. This means it’s important to know when the value is high for a long time, because it can cause the server to slow down (and become unresponsive, triggering a replica state change) or the oplog to start to lag behind.

    However, it can trigger too often, so you need to be careful. Set long delays e.g. if the lock remains above 75% for more than 30 minutes and if you have alerts on replica state and oplog lag, you can actually set this as a non-critical alert.

    Related to this is how much work your disks are doing i.e. disk i/o % utilization. Approaching 100% indicates your disks are at capacity and you need to upgrade them i.e. spinning disk to SSD. If you are using SSDs already then you can provide more RAM or you need to split the data into shards.

    MongoDB SSD performance benchmarks

    Non-critical metrics to monitor MongoDB

    There are a range of other metrics you should keep track of on a regular basis. Even though they might be non-critical, they will help avoid issues escalating to critical production problems if dealt with and investigated.

    Memory usage and page faults

    Memory is probably the most important resource you can give MongoDB and so you want to make sure you always have enough! The rule of thumb is to always provide sufficient RAM for all of your indexes to fit in memory, and where possible, enough memory for all your data too.

    Resident memory is the key metric here – MongoDB provides some useful statistics to show what it is doing with your memory.

    Page faults are related to memory because a page fault happens when MongoDB has to go to disk to find the data rather than memory. More page faults indicate that there is insufficient memory, so you should consider increasing the available RAM.

    Connections

    Every connection to MongoDB has an overhead which contributes to the required memory for the system. This is initially limited by the Unix ulimit settings but then will become limited by the server resources, particularly memory.

    High numbers of connections can also indicate problems elsewhere e.g. requests backing up due to high lock % or a problem with your application code opening too many connections.

    Shard chunk distribution

    MongoDB will try and balance chunks equally around all your shards but this can start to lag behind if there are constraints on the system e.g. high lock % slowing down moveChunk operations. You should regularly keep an eye on how balanced the cluster is.

    We have released a free tool to help with this. It can be run standalone, programmatically or as part of a plugin for Server Density.

    Tools to monitor MongoDB

    Now you know the things to keep an eye on, you need to know how to actually collect those monitoring statistics!

    Monitoring MongoDB in real time

    MongoDB includes a number of tools out of the box. These are all run against a live MongoDB server and report stats in real time:

    • mongostat – this shows key metrics like opcounts, lock %, memory usage and replica set status updating every second. It is useful for real time troubleshooting because you can see what is going on right now.
    • mongotop – whereas mongostat shows global server metrics, mongotop looks at the metrics on a collection level, specifically in relation to reads and writes. This helps to show where the most activity is.
    • rs.status() – this shows the status of the replica set from the viewpoint of the member you execute the command on. It’s useful to see the state of members and their oplog lag.
    • sh.status() – this shows the status of your sharded cluster, in particular the number of chunks per shard so you can see if things are balanced or not.

    MongoDB monitoring, graphs and alerts

    Although the above tools are useful for real time monitoring, you also need to keep track of statistics over time and get notified when metrics hit certain thresholds – some critical, some non-critical. This is where a monitoring tool such as Server Density comes in. We can collect all these statistics for you, allow you to configure alerts and dashboards and graph the data over time, all with minimal effort.

    MongoDB graphs

    If you already run your own on-premise monitoring using something like Nagios or Munin, there are a range of plugins for those systems too.

    MongoDB themselves provide free monitoring as part of the MongoDB Management Service. This collects all the above statistics with alerting and graphing, similar to Server Density but without all the other system, availability and application monitoring.

    Monitor MongoDB Slides

    Monitor MongoDB Video

  5. What’s in your on call playbook?

    Leave a Comment

    Back in February we started centralising and revamping all our ops documentation. I played around with several different tools and ended up picking Google Docs to store all the various pieces of information about Server Density, our server monitoring application.

    We make use of Puppet to manage all of our infrastructure and this acts as much of the documentation – what is installed, configuration, management of servers, dealing with failover and deploys – but there is still need for other written docs. The most important is the incident response guide, which is the step by step checklist all our on-call team run through when an alert gets triggered.

    iPhone Server Monitoring Alert

    Why do you need an incident response guide?

    As your team grows, you can’t just rely on one or two people knowing everything about how to deal with incidents in an ad-hoc manner. Systems will become more complex and you’ll want to distribute responsibilities around team members, so not everyone will have the same knowledge. During an incident, it’s important that the right things get done in the right order. There are several things to remember:

    • Log everything you do. This is important so that other responders can get up to speed and know what has been done, but is also important to review after the incident is resolved so you can make improvements as part of the postmortem.
    • Know how to communicate internally and with end-users. You want to make sure you are as efficient as possible as a team, but also keep your end-users up to date so they know what is happening.
    • Know how to contact other team members. If the first responder needs help, you need a quick way to raise other team members.

    All this is difficult to remember during the stress of an incident so what you need is an incident response guide. This is a short document that has clear steps that are always followed when an alert is triggered.

    Google Docs Incident Handling

    What should you have in your incident response guide?

    Our incident response guide contains 6 steps which I’ve detailed below, expanded upon to give some insight into the reasoning. In the actual document, they are very short because you don’t want to have complex instructions to follow!

    1. Log the incident in JIRA. We use JIRA for project management and so it makes sense to log all incidents there. We open the incident ticket as soon as the responder receives the alert and it contains the basic details from the alert. All further steps taken in diagnosing and fixing the problem are logged as comments. This allows us to refer to the incident by a unique ID, it allows other team members to track what is happening and it means we can link the incident to followup bug tasks or improvements as part of the postmortem.
    2. Acknowledge the alert in PagerDuty. We don’t acknowledge alerts until the incident is logged because we link the acknowledgment with the incident. This helps other team members know that the issue is being investigated rather than someone has accidentally acknowledged the alert and forgotten about it.
    3. Log into the Ops War Room in Hipchat. We use Hipchat for real time team communication and have a separate “war room” which is used only for discussing ongoing incidents. We use sterile cockpit rules to prevent noise and also pipe in alerts into that room. This allows us to see what is happening, sorted by timestamp. Often we will switch to using a phone call (usually via Skype because Google Hangouts still uses far too much CPU!) if we need to discuss something or coordinate certain actions, because speaking is faster than typing. Even so, we will still log the details in the relevant JIRA incident ticket.
    4. Search the incident response Google Docs folder and check known issues. We have a list of known issues e.g. debug branches deployed or known problems waiting fixes which sometimes result in on-call alerts. Most of the time though it is something unusual and we have documentation on all possible alert types so you can easily search by error string and find the right document, and the steps for debugging. Where possible we try to avoid triggering on-call alerts to real people where a problem can be fixed using an automated script, so usually these steps are debug steps to help track down where the problem is.
    5. If the issue is affecting end-users, do a post to our status site. Due to the design of our systems, we very rarely have incidents which affect the use of our product. However, where there is a problem which causes customer impact, we post to our public status page. We try and provide as much detail as possible and post updates as soon as we know more, or at the very least every 30m even if there is nothing new to report. It seems counter-intuitive that publicising your problems would be a good thing, but customers generally respond well to frequent updates so they know when problems are happening. This is no excuse for problems happening too frequently but when they do happen, customers want to know.
    6. Escalate the issue if you can’t figure it out. If the responder can’t solve the issue then we prefer they bring in help sooner rather than prolong the outage. This is either by escalating the alert to the secondary on-call in PagerDuty or by calling other team members directly.

    Replying to customer emails

    Another note we have is regarding support tickets that come in reporting the issue. Inevitably some customers are not aware of your public status page and they’ll report any problems directly to you. We use Zendesk to set the first ticket as a “Problem” and direct the customer to our status page. Any further tickets can be set as “Incidents” of that “Problem” so when we solve the issue, we can do a mass reply to all linked tickets. Even though they can get the same info from the status page, it’s good practice to email customers too.

    What do you have in your playbook?

    Every company handles incidents differently. We’ve built this process up over the years of experience, learning how others do things and understanding our own feelings when services we use have outages. You can do a lot to prevent outages but you can never eliminate them, so you need to spend as much time planning the process for handling them. What do you have in your incident response processes? Leave a comment!

  6. Cloud location matters – latency, privacy, redundancy

    Comments Off on Cloud location matters – latency, privacy, redundancy

    This article was originally published on GigaOm.

    Now that we’re seeing intense competition in the cloud infrastructure market, each of the vendors is looking for as many ways to differentiate itself as possible. Big wallets are required to build the infrastructure and picking the right locations to deploy that capital is becoming an important choice. Cloud vendors can be innovative on a product or technical level, but location is just as important — which geographies does your cloud vendor have data centers in and why does that matter?

    Why is location important?

    There are a number of reasons why a diverse range of locations is important:

    • Redundancy: Compared to the chances of a server failure, whole data center outages are rare — but they can happen. In the case of power outages, software bugs or extreme weather, it’s important to be able to distribute your workloads across multiple, independent facilities. This is not just to get redundancy across data centers but also across geographies so you can avoid local issues like bad weather or electrical faults. You need data centers close enough to minimize latency but far enough to be separated by geography.
    • Data protection: Different types of data have different locality requirements e.g. requiring personal data to remain within the EU.
    • User latency: response times for the end user are very important in certain applications, so having data centers close to your users is important, and the ability to send traffic to different regions helps simplify this. CDNs can be used for some content but connectivity is often required to the source too.

    Deploying data centers around the world is not cheap, and this is the area where the big cloud providers have an advantage. It is not just a case of equipping and staffing data centers — much of the innovation is coming from how efficient those facilities are. Whether that means using the local geography to make data centers green, or building your own power systems, this all contributes to driving down prices, which can only truly be done at scale.

    How do the top providers perform?

    The different providers all have the concept of regions or data centers within a specific geography. Usually, these are split into multiple regions so you can get redundancy within the region, but this is not sufficient for true redundancy because the whole region could fail, or there could be a local event like a storm. Therefore, counting true geographies is important:

    Cloud provider locations

    Azure is in the lead with 12 regions followed by Softlayer (10), Amazon (8) and Rackspace (6). Google loses out, with only 3 regions.

    Where is the investment going?

    It’s somewhat surprising that Amazon has gone for so long with only a single region in Europe — although this may be about to change with evidence of a new region based in Germany. If you want redundancy then you really need at least 2 data centers nearby, otherwise latency will pose a problem. For example, replicating a production database between data centers will experience higher latency if you have to send data across the ocean (from the U.S. to Ireland, say). It’s much better to replicate between Ireland and Germany!

    AWS Map

    Softlayer is also pushing into other regions with the $1.2 billion investment it announced for new data centers in 2014. Recently it launched Hong Kong and London data centers, with more planned in North America (2), Europe (2), Brazil, UAE, India, China, Japan and Australia (2).

    Softlayer network map

    The major disappointment is Google. It’s spending a lot of money on infrastructure and actually have many more data centers worldwide than are part of Google Cloud – in USA (6), Europe (3) and Asia (2) – which would place it second behind Microsoft. Of course, Google is a fairly new entrant into the cloud market and most of its demand is going to be from products like search and Gmail, where consumer requirements will dominate. Given the speed at which it’s launching new features, I expect this to change soon if it’s really serious about competing with the others.

    Google data center locations

    What about China?

    I have specifically excluded China from the figures above but it is still an interesting case. The problem is that while connectivity inside China is very good (in some regions), crossing the border can add significant latency and packet loss. Microsoft and Amazon both have regions within China, but they require a separate account and you usually have to be based in China to apply. Softlayer has announced a data center in Shanghai, so it will be interesting to see whether it can connect their global private network with good throughput. As for Google, it publicly left China 4 years ago so it may never launch a region there.

    It’s clear that location is going to be a competitive advantage, one where Microsoft currently holds first place but will lose it to Softlayer soon. Given the amount of money being invested, it will be interesting to see where cloud availability expands to next.

  7. How to monitor Nginx

    4 Comments
    Update: We hosted a live Hangout on Air with Rick Nelson the Technical Solutions architect from NGINX, in which we dug deeper into some of the issues discussed in this blog post. We’ve made the slides and video available, which can be found embedded at the bottom of this blog post.



    Nginx is a popular web server which is often used as a load balancer because of its performance. It is used extensively at Server Density to power our public facing UI and APIs, and also for its support for WebSockets. As such, monitoring Nginx is important because it is often the critical component between your users and your service.

    Monitor Nginx from the command line

    Monitoring Nginx in real time has advantages when you are trying to debug live activity or monitor what traffic is being handled in real time. These methods make use of the Nginx logging to parse and display activity as it happens.

    Enable Nginx access logging

    For monitoring the real time Nginx traffic, you first need to enable access logging by editing your Nginx config file and adding the access_log directive. As a basic example:

    
    server {
        access_log /var/log/nginx/access_log combined;
        ...
    }
    

    Then restart Nginx and tail the log as requests hit the server to see them in real time:

    
    tail -f /var/log/nginx/access_log
    

    Using ngxtop to parse the Nginx access log

    Whilst tailing the access log directly is useful for checking a small number of requests, it quickly becomes unusable if you have a lot of traffic. Instead, you can use a tool like ngxtop to parse the log file for you, displaying useful monitoring stats on the console.

    
    $ ngxtop
    running for 411 seconds, 64332 records processed: 156.60 req/sec
    
    Summary:
    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |---------+------------------+-------+-------+-------+-------|
    |   64332 |         2775.251 | 61262 |  2994 |    71 |     5 |
    
    Detailed:
    | request_path                             |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |------------------------------------------+---------+------------------+-------+-------+-------+-------|
    | /abc/xyz/xxxx                            |   20946 |          434.693 | 20935 |     0 |    11 |     0 |
    | /xxxxx.json                              |    5633 |         1483.723 |  5633 |     0 |     0 |     0 |
    | /xxxxx/xxx/xxxxxxxxxxxxx                 |    3629 |         6835.499 |  3626 |     0 |     3 |     0 |
    | /xxxxx/xxx/xxxxxxxx                      |    3627 |        15971.885 |  3623 |     0 |     4 |     0 |
    | /xxxxx/xxx/xxxxxxx                       |    3624 |         7830.236 |  3621 |     0 |     3 |     0 |
    | /static/js/minified/utils.min.js         |    3031 |         1781.155 |  2104 |   927 |     0 |     0 |
    | /static/js/minified/xxxxxxx.min.v1.js    |    2889 |         2210.235 |  2068 |   821 |     0 |     0 |
    | /static/tracking/js/xxxxxxxx.js          |    2594 |         1325.681 |  1927 |   667 |     0 |     0 |
    | /xxxxx/xxx.html                          |    2521 |          573.597 |  2520 |     0 |     1 |     0 |
    | /xxxxx/xxxx.json                         |    1840 |          800.542 |  1839 |     0 |     1 |     0 |
    

    For more long running monitoring of the logs, Luameter is a better tool that has improved performance for long running monitoring.

    Nginx monitoring and alerting – Nginx stats

    The above tools are useful for monitoring manually but aren’t useful if you want to automatically collect Nginx monitoring statistics and configure alerts on them. Nginx alerting is useful for ensuring your web server availability and performance remains high.

    The basic Nginx monitoring stats are provided by HttpStubStatusModule – metrics include requests per second and number of connections, along with stats for how requests are being handled.

    Server Density supports parsing the output of this module to automatically graph and trigger alerts on the values, so we have a guide to configuring HttpStubStatusModule too. Using this module you can keep an eye on the number of connections to your server, and the requests per second throughput. What values these “should” be will depend on your application and hardware.

    nginx monitoring alerts

    A good way to approach configuring Nginx alerts is to understand what kind of baseline traffic your application experiences and set alerts around this e.g. alert if the stats are significantly higher (indicating a sudden traffic spike) and if the values are suddenly significantly lower (indicating a problem preventing traffic somewhere). You could also benchmark your server to find out at what traffic level things start to slow down and the server becomes too overloaded – this will then act as a good upper limit which you can trigger alerts at too.

    Nginx monitoring and alerting – server stats

    Monitoring Nginx stats like requests per second and number of connections is useful to keep an eye on Nginx itself, but its performance will also be affected by how overloaded the server is. Ideally you will be running Nginx on its own dedicated instance so you don’t need to worry about contention with other applications.

    Web servers are generally limited by CPU and so your hardware spec should offer the web server as many CPUs and/or cores as possible. As you get more traffic then you will likely see the CPU usage increase.

    CPU % usage itself is not necessarily a useful metric to alert on because the values tend to be per CPU or per core. It’s more useful to set up monitoring on average CPU utilisation across all CPUs or cores. Using a tool such as Server Density, you can visualise this and configure alerts so you can be notified when the CPU is overloaded – our guide to understanding these metrics and configuring CPU alerts will help.

    On Linux this average across all CPUs is abstracted out to another system metric called load average. It is a decimal number rather than a percentage and allows you to understand load from the perspective of the operating system i.e. how long processes are waiting for access to the CPU. The recommended threshold for load average therefore depends on how many CPUs and cores you have – our guide to load average will help you understand this further.

    Monitoring Nginx and load balancers with Nginx Plus

    If you purchase a commercial version of Nginx then you get access to more advanced monitoring (and other features) without having to recompile Nginx with the HttpStubStatusModule enabled.

    Nginx Plus includes monitoring stats for connections, requests, load balancer counts, upstream metrics, the status of different load balancer upstreams and a range of other metrics. A live example of what this looks like is provided by Nginx themselves. It also includes a JSON Nginx monitoring API which would be useful for pulling the data out into your own tools.

    Monitoring nginx in real time

    Monitoring the remote status of Nginx

    All of the above metrics monitor the internal status of Nginx and the servers it is running on but it is also important to monitor the experience your users are getting too. This is achieved by using external status and response time tools – you want to know if your Nginx instance is serving traffic from different locations around the world (wherever your customers are) and the kind of response time performance.

    This is easy to do with a service like Server Density because of our in-built website monitoring. You can check the status of your public URLs and other endpoints from custom locations and get alerts when performance drops or there is an outage.

    This is particularly useful when you can build graphs to correlate the Nginx and server metrics with remote response time, especially if you are benchmarking your servers and want to know when a certain load average starts to affect end user performance.

    Monitor Nginx Slides

    Monitor Nginx Video

  8. Sysadmin Sunday 189

    Comments Off on Sysadmin Sunday 189
  9. How we use Puppet – infrastructure, config, failover, deploys

    Comments Off on How we use Puppet – infrastructure, config, failover, deploys

    We’ve been using Puppet to manage the infrastructure behind Server Density for several years now. It helps us in a number of different ways and although we use it as standard config management, that’s actually only about 25% of our use case.

    We have 4 main uses for Puppet – infrastructure, config management, failover and deploys – each of which I’ll go through here.

    How we use Puppet – Infrastructure

    We first started using Puppet when we moved our environment to Softlayer, where we have a mixture of bare metal servers and public cloud instances, totalling around 75-100. When this was set up, we ordered the servers from Softlayer then manually installed Puppet before applying our manifests to get things configured.

    Although we recently evaluated moving to running our own environment in colo data centres, we have made the decision to switch our environment from Softlayer to Google Cloud. My general view remains that colo is significantly cheaper in the long run but there are some initial capital expenses which we don’t want to spend. We also want to make use of some of the Google products like BigQuery.

    I’ll be writing about this in more detail as we complete the move and am also giving two talks – one in the Bay Area on the 19th June and the other in London on the 2nd of July – about this.

    Using Google Cloud (specifically, Google Compute Engine), or indeed any one of the other major cloud providers, means we can make use of Puppet modules to define the resources within our code. Instead of having to manually order them through the control panels, we can define them in the Puppet manifests alongside the configuration. We’re using the gce_compute module but there are also modules for Amazon and others.

    For example, defining an instance plus a 200GB volume:

    gce_instance { 'mms-app1':
      ensure         => present,
      machine_type   => 'n1-highmem-2',
      zone           => 'us-central1-a',
      network        => 'private',
      tags           => ['mms-app', 'mongodb'],
      image          => 'projects/debian-cloud/global/images/backports-debian-7-wheezy-v20140605',
    }
    
    gce_disk { 'mms-app1-var-lib-mongodb':
      ensure      => present,
      description => 'mms-app1:/var/lib/mongodb',
      size_gb     => '200',
      zone        => 'us-central1-a',
    }

    The key here is that we can define instances in code, next to the relevant configuration for what’s running on them, then let Puppet deal with creating them.

    How we use Puppet – Config management

    This is the original use case for Puppet – defining everything we have installed on our servers in a single location. It makes it easy to deploy new servers and keep everything consistent.

    It also means any unusual changes, fixes or tweaks are fully version controlled and documented so we don’t lose things over time e.g. we have a range of fixes for MongoDB to work around issues and make optimisations which have been built up over time and through support requests, all of which are documented in Puppet.

    Puppet in Github

    We use the standard module layout as recommended by Puppet Labs, contained within a Github repo and checked with puppet lint before commit so we have a nicely formatted, well structured library describing our setup. Changes go through our usual code review process and get deployed with the Puppet Master picking up the changes and rolling them out.

    Previously, we wrote our own custom modules to describe everything but more recently where possible we use modules from the Puppet Forge. This is because they often support far more options and are more standardised than our own custom modules. For example, the MongoDB module allows us to install the server and client, set options and even configure replica sets.

      include site::mongodb_org
    
      class {'::mongodb::server':
        ensure    => present,
        bind_ip   => '',
        replset   => 'mms-app',
      }
    
      mount {'/var/lib/mongodb':
        ensure  => mounted,
        atboot  => true,
        device  => '/dev/sdb',
        fstype  => 'ext4',
        options => 'defaults,noatime',
        require => Class['::mongodb::server'],
      }
    
      mongodb_replset { 'mms-app':
        ensure  => present,
        members => ['mms-app1:27017', 'mms-app2:27017', 'mms-app3:27017']
      }
    

    We pin specific versions of packages to ensure the same version always gets installed and we can control upgrades. This is particularly important to avoid sudden upgrades of critical packages, like databases!

    The Server Density monitoring agent is also available as a Puppet Forge module to automatically install the agent, register it and even define your alerts.

    All combined, this means we have our MongoDB backups running on Google Compute Engine, deployed using Puppet and monitored with Server Density.

    MMS

    How we use Puppet – Failover

    We use Nginx as a load balancer and use Puppet variables to list the members of the proxy pool. This is deployed using a Puppet Forge nginx module we contributed some improvements to.

    When we need to remove nodes from the load balancer rotation, we can do this using the Puppet web UI as a manual process, or by using the console rake API. The UI makes it easy to apply the changes so a human can do it with minimal chance of error. The API allows us to automate failover in particular conditions, such as if one of the nodes fails.

    How we use Puppet – Deploys

    This is a more unusual way of using Puppet but has allowed us to concentrate on building a small portion of the deployment mechanism, taking advantage of the Puppet agent which runs on all our servers already. It saves us having to use custom SSH commands or writing our own agent, and allows us to customise the deploy workflow to suit our requirements.

    It works like this:

    1. Code is committed in Github into master (usually through merging a pull request, which is how we do our code reviews).
    2. A new build is triggered by Buildbot which runs our tests, then creates the build artefacts – the stripped down code that is actually copied to the production servers.
    3. Someone presses the deploy button in our internal control panel, choosing which servers to deploy to (branches can also be deployed) and the internal version number is updated to reflect what should be deployed).
    4. /opt/puppet/bin/mco puppetd runonce -I is triggered on the selected hosts and the Puppet run notices that the deployed version is different from the requested version.

    5. The new build is copied onto the servers.

    Buildbot

    Status messages are posted into Hipchat throughout the process and any one of our engineers can deploy code at any time, although we have a general rule not to deploy non-critical changes after 5pm weekdays and after 3pm on Fridays.

    There are some disadvantages to using Puppet for this. Firstly, the Puppet agent can be quite slow on low spec hardware. Our remote monitoring nodes around the world are generally low power nodes so the agent runs very slowly. It’s also eventually consistent because deploys won’t necessarily happen at the same time, so you need to account for that in new code you deploy.

    Puppet is most of your documentation

    These four use cases mean that a lot of how our infrastructure is set up and used is contained within text files. This has several advantages:

    • It’s version controlled – everyone can see changes and they are part of our normal review process.
    • Everyone can see it – if you want to know how something works, you can read through the manifests and understand more, quickly.
    • Everything is consistent – it’s a single source of truth, one place where everything is defined.

    It’s not all of our docs, but certainly makes up a large proportion because it’s actually being used, live. And everyone knows how much they hate keeping the docs up to date!

    Puppet is the source of truth

    It knows all our servers, where they are, their health status, how they are configured and what is installed.

  10. How to keep on top of security updates

    2 Comments

    Every computer user needs to stay on top of updates for their apps, browsers, and OSs. As a consumer, it’s easy to do this – the best browsers like Chrome auto update in the background and over the air updates on iOS make sure most people get updates quickly, and easily. These kind of mass market updates tend to be well tested so you can update without fear of bricking.

    The life of a sysadmin is a little more complex though. Although the mainstream OSs provide mature package management features, there are often a lot of frequent updates across the OS, kernel updates which require reboots, application (e.g. databases) updates which need testing and library release which need recompiling or updating codebases.

    I’ve written before about our canary concept for rolling out updates because we deploy them manually rather than letting the OS auto update whenever it likes, but how do you discover new releases in the first place?

    OS X Update

    Critical security update notifications

    The first step is to separate general OS/library/app updates with critical security releases. Like the recent Heartbleed bug, these need to be deployed as quickly as possible.

    The first point of call is the OS security announcements. We use Ubuntu and so have the choice of subscribing to the Ubuntu Security Announcements list, or via RSS. In these announcements I look out for keywords like “remote” or “denial of service” because these mean that there’s an external risk, contrasted to an exploit which requires a local user. This is more difficult to exploit because it first requires access to our servers, which is restricted to our ops team.

    Debian has a similar mailing list as do other Linux distros like Red Hat. Windows has an update mechanism built in which makes it easier than with multiple Linux distros.

    Ubuntu updates

    Updating software packages

    Most of the software products and libraries we use are installed through system packages using apt. This is our preferred method of installing because it makes it easy to manage via Puppet, is standardised across all our servers, offers signing of packages and importantly, gives us a single place to apply upgrades. This is one of the reasons why our monitoring agent recommended installation method is via our OS packages.

    A lot of packages are provided as part of the OS release from Ubuntu package maintainers, so they get rolled into the security notifications.

    The disadvantage of this approach is that you often don’t get the very latest version through the OS vendor packages. To work around this, vendors will usually provide their own official repositories (e.g. our MongoDB installations are through the MongoDB, Inc repos).

    apt-get upgrade

    Everything else is manual

    A few things are installed manually, such as PHP pecl packages for some of our legacy apps. We can update them using pecl but have to keep an eye on the release mailing lists for those specific packages.

    And where we run “on-premise” versions of products we also subscribe to the relevant mailing lists. Our policy is to prefer SaaS where possible but some things aren’t available like this – we run our own Puppet Enterprise server (with an announce mailing list) and have the MongoDB Backup Service running within our infrastructure.

    We also keep an eye on pages like the MongoDB Alerts and mailing list.

    puppet-announce

    What about general security mailing lists?

    You could subscribe to mailing lists like BUGTRAQ or the CVE lists but these are really high volume and aren’t specific enough to our environment. We can get what we need from the vendors, mostly from Ubuntu.

    There are also commercial products like the Ubuntu Landscape product which we used for a while but it was too expensive to maintain a support contract for so many servers, with limited value. So long as we can stay up to date with the most important security releases, we can be sure that we have a secure infrastructure as regards to software updates.