Premium Hosted Website & Server Monitoring Tool.

(Sysadmin / Devops blog)

visit our website

Author Archives: David Mytton

About David Mytton

David Mytton is the founder of Server Density. He has been programming in PHP and Python for over 10 years, regularly speaks about MongoDB (including running the London MongoDB User Group), co-founded the Open Rights Group and can often be found cycling in London or drinking tea in Japan. Follow him on Twitter and Google+.
  1. Network performance at AWS, Google, Rackspace and Softlayer

    Leave a Comment

    This post was originally published on GigaOm.

    Compute and storage are essentially commodity services, which means that for cloud providers to compete, they have to show real differentiation. This is often achieved with supporting services like Amazon’s DynamoDB and Route 53, or Google’s BigQuery and Prediction API, which complement the core infrastructure offerings.

    Performance is also often singled out as a differentiator. Often one of the things that bites production usage, especially in inherently shared cloud environments, is the so-called “noisy neighbor” problem. This can be other guests stealing CPU time, increased network traffic and, particularly problematic for databases, i/o wait.

    In this post I’m going to focus on networking performance. This is very important for any serious application because it affects the ability to communicate and replicate data across instances, zones and regions. Responsive applications and disaster recovery, areas where up-to-date database replication is critical, require good, consistent performance.

    It’s been suggested that Google has a massive advantage when it comes to networking, due to all the dark fibre it has purchased. Amazon has some enhanced networking options that take advantage of special instance types with OS customizations, and Rackspace’s new Performance instance types also boast up to 10 Gbps networking. So let’s test this.

    Methodology

    I spun up the listed instances to test the networking performance between them. This was done using the iperf tool on Linux. One server acts as the client and the other as the server:

    Server: iperf -f m -s
    Client: iperf -f m -c hostname

    The OS was Ubuntu 12.04 (with all latest updates and kernel), except on Google Compute Engine, where it’s not available. There, I used the Debian Backports image.

    The client was run for three tests for each type – within zone, between zones and between regions – with the mean average taken as the value reported.

    Amazon networking performance

    Amazon networking performance

    Amazon’s larger instances, such as the c3.8xlarge tested here, support the enhanced 10 GB networking, however you must use the Amazon Linux AMI (or manually install the drivers) within a VPC. Because of the additional complexity of setting up a VPC, which isn’t necessary on any other provider, I didn’t test this, although it is now the default for new accounts. Even without that enhancement, the performance is very good, nearing the advertised 10 Gbits/sec.

    However, the consistency of the performance wasn’t so good. The speeds changed quite dramatically across the three test runs for all instance types, much more than with any other provider.

    You can use internal IPs within the same zone (free of charge) and across zones (incurs inter-zone transfer fees), but across regions, you have to go over the public internet using the public IPs, which incurs further networking charges.

    Google Compute Engine networking performance

    Google networking performance

    Google doesn’t currently offer an Ubuntu image, so instead I used its backports-debian-7-wheezy-v20140318 image. For the f1-micro instance, I got very inconsistent iperf results for all zone tests. For example, within the same us-central-1a zone, the first run showed 991 Mbits/sec, but the next two showed 855 Mbits/sec and 232 Mbits/sec. Across regions between the US and Europe, the results were much more consistent, as were all the tests for the higher spec n1-highmem-8 server. This suggests the variability was because of the very low spec, shared CPU f1-micro instance type.

    I tested more zones here than on other providers because on April 2, Google announced a new networking infrastructure in us-central-1b and europe-west-1a which would later roll out to other zones. There was about a 1.3x improvement in throughput using this new networking and users should also see lower latency and CPU overhead, which are not tested here.

    Although 16 CPU instances are available, they’re only offered in limited preview with no SLA, so I tested on the fastest generally available instance type. Since networking is often CPU bound, there may be better performance available when Google releases its other instance types.

    Google allows you to use internal IPs globally — within zone, across zone and across regions (i.e., using internal, private transit instead of across the internet). This makes it much easier to deploy across zones and regions, and indeed Google’s Cloud platform was the easiest and quickest to work with in terms of the control panel, speed of spinning up new instances and being able to log in and run the tests in the fastest time.

    Rackspace networking performance

    Rackspace networking performance

    Rackspace does not offer the same kind of zone/region deployments as Amazon or Google so I wasn’t able to run any between-zone tests. Instead I picked the next closest data center. Rackspace offers an optional enhanced virtualization platform called PVHVM. This offers better i/o and networking performance and is available on all instance types, which is what I used for these tests.

    Similar to Amazon, you can use internal IPs within the same location at no extra cost but across regions you need to use the public IPs, which incur data charges.

    When trying to launch x2 120 GB Performance 2 servers at Rackspace, I hit our account quota (with no other servers on the account) and had to open a support ticket to request a quota increase, which took them about an hour and a half to approve. For some reason, launching servers in the London region also requires a separate account, and logging in and out of multiple control panels soon became annoying.

    Softlayer networking performance

    Softlayer networking performance

    Softlayer only allows you to deploy into multiple data centers at one location: Dallas. All other regions have a single facility. Softlayer also caps out at 1 Gbps on its public cloud instances, although its bare metal servers do have the option of dual 1 Gbps bonded network cards, allowing up to 2 Gbps. You choose the port speed when ordering or when upgrading an existing server. They also list 10Gbit/s networking as available for some bare metal servers.

    Similarly to Google, Softlayer’s maximum instance size is 16 cores, but it also offers private CPU options which give you a dedicated core versus sharing the cores with other users. This allows up to eight private cores, for a higher price.

    The biggest advantage Softlayer has over every other provider is completely free, private networking between all regions whereas all other provider charge for transfer out of zone. When you have VLAN spanning enabled, you can use the private network across regions, which gives you an entirely private network for your whole account. This makes it very easy to deploy redundant servers across regions and is something we use extensively for replicating MongoDB at Server Density, moving approx 500 Mbits/sec of internal traffic across the US between Softlayer’s Washington and San Jose data centers. Not having to worry about charges is a luxury only available with Softlayer.

    Who is fastest?

    Who is fastest?

    Amazon’s high spec c3.8xlarge really gives the best performance across all tests, particularly within the same zone and region. It was able to push close to the advertised 10 GB throughput, but the high variability of results may indicate some inconsistency in the real-world performance.

    Yet for very low cost, Google’s low spec f1-micro instance type offers excellent networking performance: ten times faster than the terrible performance from the low spec Rackspace server.

    Softlayer and Rackspace were generally bad performers overall, but at least Rackspace gets some good inter-zone and inter-region performance and performed well for its higher instance spec. Softlayer is the loser overall here with low performance plus no network-optimized instance types. Only their bare metal servers have the ability to upgrade to 10 Gbits/sec network interfaces.

    Mbits/s per CPU?

    CPU allocation is also important. Rackspace and Amazon both offer 32 core instances, and we see good performance on those higher spec VMs as a result. Amazon was fastest for its highest spec machine type with Rackspace coming second. The different providers have different instance types, and so it’s difficult to do a direct comparison on the raw throughput figures.

    An alternative ranking method is to calculate how much throughput you get per CPU. We’ll use the high spec inter-zone figures and do a simple division of the throughput by the number of CPUs:

    Mbits per CPU

    The best might not be the best value

    If you have no price concerns, then Amazon is clearly the fastest, but it’s not necessarily the best value for money. Google gets better Mbits/sec per CPU performance, and since you pay for more CPUs, it’s a better value. Google also offers the best performance on its lowest spec instance type, but it is quite variable due to the shared CPU. Rackspace was particularly poor when it came to inter-region transfer, and Softlayer isn’t helped by its lack of any kind of network-optimized instance types.

    Throughput isn’t the end of the story though. I didn’t look at latency or CPU overhead and these will have an impact on the real world performance. It’s no good getting great throughput if it requires 100 percent of your CPU time!

    Google and Softlayer both have an advantage when it comes to operational simplicity because their VLAN spanning-like features mean you have a single private network across zones and regions. You can utilize their private networking anywhere.

    Finally, pricing is important, and an oft-forgotten cost are the network transfer fees. This is free within zones for all providers, but only Softlayer has no fees for data transfer across zones and even across regions. This is a big saver.

  2. Android Server Monitoring App released

    Leave a Comment

    After the release of our monitoring app for iPhone 2 months ago, the same app is now available on Android!

    It allows you to see your notification center on the move so you can quickly see open and closed alerts assigned to you and across your whole account. It includes push notification alerts and custom sounds so you can always be woken up!

    Android Server Monitoring App Use Case

    The app is free for all v2 customers (and users on the trial). Find out more on the product page of our website.

  3. How we use Hipchat to work remotely

    2 Comments

    Server Density started as a remote working company, and it wasn’t until our 3rd year that we opened an office in London. Even now, most of the team still work remotely from around the UK, Portugal and Spain so we use a variety of tools to help us stay connected and work efficiently together.

    One of the most important is Hipchat. We use this as a chat room but it’s also our company news feed. Everything that goes on during the day (and night) gets posted and we use a variety of rooms for different purposes.

    The main room

    Everyone is always logged in here during their working day and it also acts as a real time news feed of what is going on. General chat happens here but it’s mostly a way for everyone to see what’s happening. This is particularly useful if you go away and come back later because you can see what has happened.

    Main Hipchat room

    We use various integrations and the Hipchat API to post in events from things like:

    • Github activity: commits, pull requests, comments.
    • Buildbot: Build activity.
    • Deploys: From our own custom build system, we can see when deploys are triggered (and by whom) and then when they get deployed to each server.
    • Signups: New trial signups get posted in as we often like to see who is using the product.
    • Account changes: New purchases and package changes
    • JIRA: All our issue tracking and development work is tracked with JIRA, which posts some activity like new issues and status changes.
    • Zendesk: New support tickets and Twitter mentions so everyone can keep an eye on emerging issues.
    • Alerts: We use some of our competitors to help us maintain an independent monitor of our uptime, and pipe in alerts using the new Server Density HipChat integration and PagerDuty.

    Ops war room

    All incidents from our alerting systems get piped into a war room so we can see what’s going on in real time, chronological order and not be distracted by other events. The key here is maintaining a sterile cockpit rule so we use this room to only discuss ongoing incidents. This also is useful for other people (e.g. support) to track what’s happening without interrupting the responders.

    Server Density hipchat

    Bot

    We have a bot that runs in all of our chat rooms. It’s fairly simple and based off Hubot but allows us to do things like query Google Images or check the status of Github.

    Bot

    At a glance

    We use our own server monitoring ops dashboard on the TV in the office using a Chromecast and I also have an iPad at my desk which constantly shows the Hipchat room, so I can see things as they happen throughout the day.

    Server Density office

  4. Dealing with OpenSSL bug CVE-2014-0160 (Heartbleed)

    2 Comments

    Yesterday a serious vulnerability in the OpenSSL library was disclosed in CVE-2014-0160, also dubbed the Heartbleed Vulnerability. Essentially this means you probably need to regenerate the private keys used to create your SSL certificates, and have them reissued by your certificate authority.

    Heartbleed

    This isn’t a difficult task but does take some time to get OpenSSL updated across all your servers, then go through the process to generate, reissue and install certificates across all locations they are deployed.

    We have completed this process for all of our websites and applications and for Server Density v2 we use perfect forward security which should protect against retrospective decryption of previous communications.

    However, in the latest release of our server monitoring iPhone app we enabled certificate pinning which means until our latest update is approved by Apple, the app will not log in. You will still receive push notifications for alerts but attempts to log in to the app will fail. Certificate pinning embeds our SSL certificate within the app which is then used to prevent man in the middle attacks – the certificate that is returned through the API calls to our service is verified against the known certificate embedded in the app.

    We discussed the best way to approach the reissue of certificates this morning and considered holding off to allow us to submit a new build to Apple with pinning disabled, then do a future update with the new certificate. However, we felt that the security vulnerability was severe enough that we should patch it for all our users at the expense of causing a small number of users to be unable to use the iPhone app for a few days.

    We have requested Apple expedite the review process but it still takes at least 24 hours to get a new release out. In the meantime, you should check to see if your OpenSSL version is vulnerable and if so, update!

  5. MongoDB on Google Compute Engine – tips and benchmarks

    6 Comments

    Over the last 4 years running MongoDB in production at Server Density, I’ve been able to work on deployments on dedicated hardware, VMs and across multiple cloud providers.

    The best environment has always been dedicated servers because of the problems with host contention, particularly with CPU and disk i/o but Google has been quite vocal about the consistency and performance of their Compute Engine product, particularly about how they’ve eliminated the noisy neighbour problem with intelligent throttling.

    So I thought I’d try it out to see how MongoDB performs on Google Compute Engine.

    Google Compute Engine Logo

    Testing the Write Concern – performance vs durability

    The MongoDB Write Concern is historically controversial because Mongo was originally designed to get very high write throughput at the expense of durability but this wasn’t well documented. The default was changed a while back and it now gives you an acknowledgement that a write was accepted, but is still quite flexible to allow you to tune whether you want speed or durability.

    I am going to test a range of write concern options to allow us to see what kind of response times we can expect:

    Unacknowledged: w = 0 AND j = 0

    This is a fire and forget write where we don’t know if the write was successful and detection of things like network errors is uncertain.

    Acknowledged: w = 1 AND j = 0 (the default)

    This will give us an acknowledgment that the write was successfully received but no indication that it was actually written. This picks up most errors e.g. parse errors, network errors etc.

    Journaled: w = 1 AND j = 1

    This will cause the write to wait until it has been both acknowledged and written to the journal of the primary replica set member. This gives you single server durability but doesn’t guarantee the data has been replicated to other members of your cluster. In theory you could have a data center failure and lose the write.

    Replica acknowledged: w = 2 AND j = 0

    The test will give us an idea how long it takes for the write to be acknowledged on the replica set primary and acknowledged by at least 1 other member of the replica set. This gives us some durability across 2 servers but in theory the write could still fail on both because we are not doing a check for the write hitting the journal.

    Replica acknowledged and Journaled: w = 2 AND j = 1

    This ensures that the write is successfully written to the primary and has been acknowledged by at least one of the replica set members.

    Replica acknowledged with majority: w = majority AND j = 0

    In a multi datacenter environment you want to know that your writes are safely replicated. Using the majority keyword will allow you to be sure that the write has been acknowledged on the majority of your replica set members. If you have the set deployed evenly across data centers then you know that your data is safely in multiple locations.

    Replica acknowledged with majority and journaled: w = majority AND j = 1

    Perhaps the most paranoid mode, we will know that the write was successfully acknowledged by the primary and was replicated to a majority of the nodes.

    Environment configuration

    Replica sets

    Real world applications use replica sets to give them redundancy and failover capabilities across multiple data centers. To accurately test this, the test environment will involve 4 data nodes across 2 zones: x2 in the us-central1-a zone and x2 in the us-central1-b zone.

    In a real deployment you must have a majority to maintain the set in the event of a failure, so we should deploy a 5th node as an arbiter in another data center. I’ve not done this here for simplicity.

    Google Compute Engine

    I tested with the n1-standard-2 (2 vCPUs and 7.5GB RAM) and n1-highmem-8 (8 vCPUs and 52GB RAM) instance types – with the backports-debian-7-wheezy-v20140318 OS image.

    Be aware that the number of CPU cores your instance has also affects the i/o performance. For maximum performance then you need to use the 4 or 8 core instance types even if you don’t need all the memory they provide.

    There is also a bug in the GCE Debian images where the default locale isn’t set. This prevents MongoDB from starting properly from the Debian packages. The workaround is to set a default:

    sudo locale-gen en_US.UTF-8
    sudo dpkg-reconfigure locales

    Google Persistent Disks

    It’s really important to understand the performance characteristics of Google Persistent Disks and how IOPs scale linearly with volume size. Here are the key things to note:

    • At the very least you need to mount your MongoDB dbpath on a separate persistent disk. This is because the default root volume attached to every Compute Engine instance is very small and will therefore have poor performance. It does allow bursting for the OS but this isn’t sufficient for MongoDB which will typically have sustained usage requirements.
    • Use directoryperdb to give each of your databases their own persistent disk volume. This allows you to optimise both performance and cost because you can resize the volumes as your data requirements grow and/or to gain the performance benefits of more IOPs.
    • Putting the journal on a separate volume is possible even without directoryperdb because it is always in its own directory. Even if you don’t put your databases on separate volumes, it is worth separating the journal onto its own persistent disk because the performance improvements are significant – up to x3 (see results below).
    • You may be used to only needing a small volume for the journal because it uses just a few GB of space. However, allocating a small persistent disk volume will mean you get poor performance because the available IOPs increase with volume size. Choose a volume of at least 200GB for the journal.
    • If you split all your databases (or even just the journal) onto different volumes then you will lose the ability to use snapshots for backups. This is because the snapshot across multiple volumes won’t necessarily happen at the same time and will therefore be inconsistent. Instead you will need to shut down the mongod (or fsync lock it) and then take the snapshot across all disks.

    I’ve run the testing several times with different disk configurations so I can see the different performance characteristics:

    1. With no extra disks i.e. dbpath on the default 10GB system volume
    2. With a dedicated 200GB persistent disk for the dbpath
    3. With a dedicated 200GB persistent disk for the dbpath and another dedicated 200GB persistent disk for the journal

    Test methodology

    I wrote a short Python script to insert a static document into a collection. This was executed 1000 times and repeated 3 times. The Python timeit library was used to complete the tests so the fastest time was taken, as per the docs indicating that the mean/standard deviation of the 3 test cycles is not that useful.

    Results – MongoDB performance on Google Compute Engine

    n1-standard-2

    Test results - n1-standard-2

    n1-highmem-8

    Test results - results-n1-highmem-8

    Conclusions

    There’s not much difference in performance when you’re using a single persistent disk volume until you start increasing the durability options because acknowledged and unacknowledged writes are just going to memory. When you increase the write concern options then you become limited by the disk performance and it’s clear that splitting the journal onto a separate volume makes a significant difference.

    The increased performance of the larger n1-highmem-8 instance type with 8 cores vs 2 cores of the n1-standard-2 is also apparent in the figures – although the actual difference is quite small, it is still a difference which would likely help in a real world environment.

    The key takeaway is that performance decreases as durability increases – this is a tradeoff. For maximum performance with durability then you can take advantage of Compute Engine’s higher spec instance types and larger, separate persistent disk volumes per database / journal.

  6. What’s new in Server Density – Mar 2014

    Leave a Comment

    Each month we’ll round up all the feature changes and improvements we have made to our server and website monitoring product, Server Density.

    Server snapshot

    The new snapshot view allows you to browse back in time minute by minute to see the exact state of a server at any given time. It reveals the key metric absolute values, visualising them in an easy to read format. This is great for picking out spikes and unusual events on the graphs.

    You can access this from the Snapshot tab when viewing your servers, or by clicking on the Snapshot link from the popover in the graphs.

    Server snapshot

    Improved graph popovers

    If you have many series on the graph, the new popovers shown when hovering contain the items in a scrollable view. They’re more intelligent about where they show up on the page to avoid your mouse and by clicking the graph, you can pin them in the same position whilst moving the mouse cursor away. This makes it easier to compare across multiple graphs, especially when scrolling.

    Graph popovers

    iPhone background refreshing

    Since the release of our iPhone monitoring app last month, we’ve pushed out several minor updates to fix bugs and have added background refreshing of the alert list so it’s up to date as soon as you load the app.

    v2 Chef recipe

    Thanks to some excellent pull requests from Mal Graty at idio (one of our customers), we now have a v2 of our Chef recipe. On a basic level it will install the agent and register devices through our API but also includes full LWRP support and the ability to define your alerts in Chef so you can manage alert configuration in your code.

    See what’s changed in the pull request and visit Github for the recipe. The version number has been bumped in Berkshelf.

    Opsgenie integration

    If you’re using Opsgenie for your alerting, they now have an official guide for using our webhooks to trigger alerts.

    List search filters in URL

    If you search and/or filter your devices/services list, these will be reflected in the URL to allow you to link to the list and use your browser back button to go back to the list.

    New blog theme

    We’ve redesigned the theme for the blog to make it more minimalist and to work better on mobile devices.

    What’s next?

    For the next month, we’re working finishing the last few polishes for our Android app, Firefox support and dynamic graphing for the dashboard plus a few other things! We’ll also be adding process lists to the snapshot, although this won’t be out for a few months.

  7. 5 things you probably don’t know about Google Cloud

    2 Comments

    At the beginning of the month I wrote a post on GigaOM following some experimenting with Google Cloud: 5 things you probably don’t know about Google Cloud.

    1. Google Compute Engine Zones are probably in Ireland and Oklahoma
    2. Google’s Compute Zones may be isolated, but they’re surprisingly close together
    3. Scheduled maintenance takes zones offline for up to two weeks
    4. You cannot guarantee where your data will be located
    5. Connectivity across regions isn’t fast

    I think Google is currently in the best position to challenge Amazon because they have the engineering culture and technical abilities to release some really innovative features. IBM has bought into some excellent infrastructure at Softlayer but still has to prove its cloud engineering capabilities.

    Amazon has set the standard for how we expect cloud infrastructure to behave, but Google doesn’t conform to these standards in some surprising ways. So, if you’re looking at Google Cloud, have a read of the article.

  8. The tech behind our time series graphs – 2bn docs per day, 30TB per month

    19 Comments

    This post was also published on the MongoDB Blog.

    Server Density processes over 30TB/month of incoming data points from the servers and web checks we monitor for our customers, ranging from simple Linux system load average to website response times from 18 different countries. All of this data goes into MongoDB in real time and is pulled out when customers need to view graphs, update dashboards and generate reports.

    We’ve been using MongoDB in production since mid-2009 and have learned a lot over the years about scaling the database. We run multiple MongoDB clusters but the one storing the historical data does the most throughput and is the one I shall focus on in this article, going through some of the things we’ve done to scale it.

    1. Use dedicated hardware, and SSDs

    All our MongoDB instances run on dedicated servers across two data centers at Softlayer. We’ve had bad experiences with virtualisation because you have no control over the host, and databases need guaranteed performance from disk i/o. When running on shared storage (e.g a SAN) this is difficult to achieve unless you can get guaranteed throughput from things like AWS’s Provisioned IOPS on EBS (which are backed by SSDs).

    MongoDB doesn’t really have many bottlenecks when it comes to CPU because CPU bound operations are rare (usually things like building indexes), but what really causes problem is CPU steal – when other guests on the host are competing for the CPU resources.

    The way we have combated these problems is to eliminate the possibility of CPU steal and noisy neighbours by moving onto dedicated hardware. And we avoid problems with shared storage by deploying the dbpath onto locally mounted SSDs.

    I’ll be speaking in-depth about managing MongoDB deployments in virtualized or dedicated hardware at MongoDB World this June.

    2. Use multiple databases to benefit from improved concurrency

    Running the dbpath on an SSD is a good first step but you can get better performance by splitting your data across multiple databases, and putting each database on a separate SSD with the journal on another.

    Locking in MongoDB is managed at the database level so moving collections into their own databases helps spread things out – mostly important for scaling writes when you are also trying to read data. If you keep databases on the same disk you’ll start hitting the throughput limitations of the disk itself. This is improved by putting each database on its own SSD by using the directoryperdb option. SSDs help by significantly alleviating i/o latency, which is related to the number of IOPS and the latency for each operation, particularly when doing random reads/writes. This is even more visible for Windows environments where the memory mapped data files are flushed serially and synchronously. Again, SSDs help with this.

    The journal is always within a directory so you can mount this onto its own SSD as a first step. All writes go via the journal and are later flushed to disk so if your write concern is configured to return when the write is successfully written to the journal, making those writes faster by using an SSD will improve query times. Even so, enabling the directoryperdb option gives you the flexibility to optimise for different goals e.g. put some databases on SSDs and some on other types of disk (or EBS PIOPS volumes) if you want to save cost.

    It’s worth noting that filesystem based snapshots where MongoDB is still running are no longer possible if you move the journal to a different disk (and so different filesystem). You would instead need to shut down MongoDB (to prevent further writes) then take the snapshot from all volumes.

    3. Use hash-based sharding for uniform distribution

    Every item we monitor (e.g. a server) has a unique MongoID and we use this as the shard key for storing the metrics data.

    The query index is on the item ID (e.g. the server ID), the metric type (e.g. load average) and the time range but because every query always has the item ID, it makes it a good shard key. That said, it is important to ensure that there aren’t large numbers of documents under a single item ID because this can lead to jumbo chunks which cannot be migrated. Jumbo chunks arise from failed splits where they’re already over the chunk size but cannot be split any further.

    To ensure that the shard chunks are always evenly distributed, we’re using the hashed shard key functionality in MongoDB 2.4. Hashed shard keys are often a good choice for ensuring uniform distribution but if you end up not using the hashed field in your queries, could actually hurt performance because then a non-targeted scatter/gather query has to be used.

    4. Let MongoDB delete data with TTL indexes

    The majority of our users are only interested in the highest resolution data for a short period and more general trends over longer periods, so over time we average the time series data we collect then delete the original values. We actually insert the data twice – once as the actual value and once as part of a sum/count to allow us to calculate the average when we pull the data out later. Depending on the query time range we either read the average or the true values – if the query range is too long then we risk returning too many data points to be plotted. This method also avoids any batch processing so we can provide all the data in real time rather than waiting for a calculation to catch up at some point in the future.

    Removal of the data after a period of time is done by using a TTL index. This is set based on surveying our customers to understand how long they want the high resolution data for. Using the TTL index to delete the data is much more efficient than doing our own batch removes and means we can rely on MongoDB to purge the data at the right time.

    Inserting and deleteing a lot of data can have implications for data fragmentation but using a TTL index helps because it automatically activates PowerOf2Sizes for the collection, making disk usage more efficient. Although as of MongoDB 2.6 this storage option will become the default.

    5. Take care over query and schema design

    The biggest hit on performance I have seen is when documents grow, particularly when you are doing huge numbers of updates. If the document size increases after it has been written then the entire document has to be read and rewritten to another part of the data file with the indexes updated to point to the new location, which takes significantly more time than simply updating the existing document.

    As such, it’s important to design your schema and queries to avoid this, and to use the right modifiers to minimise what has to be transmitted over the network and then applied as an update to the document. A good example of what you shouldn’t do when updating documents is to read the document into your application, update the document, then write it back to the database. Instead, use the appropriate commands – such as set, remove, and increment – to modify documents directly.

    This also means paying attention to the BSON data types and pre-allocating documents, things I wrote about in MongoDB schema design pitfalls.

    6. Consider network throughput & number of packets

    Assuming 100Mbps networking is sufficient is likely to cause you problems, perhaps not during normal operations but probably when you have some unusual event like needing to resync a secondary replica set member.

    When cloning the database, MongoDB is going to use as much network capacity as it can to transfer the data over as quickly as possible before the oplog rolls over. If you’re doing 50-60Mbps of normal network traffic, there isn’t much spare capacity on a 100Mbps connection so that resync is going to be held up by hitting the throughput limits.

    Also keep an eye on the number of packets being transmitted over the network – it’s not just the raw throughput that is important. A huge number of packets can overwhelm low quality network equipment – a problem we saw several years ago at our previous hosting provider. This will show up as packet loss and be very difficult to diagnose.

    Conclusions

    Scaling is an incremental process – there’s rarely one thing that will give you a big win. All of these tweaks and optimisations together help us to perform thousands of write operations per second and get response times within 10ms whilst using a write concern of 1.

    Ultimately, all this ensures that our customers can load the graphs they want incredibly quickly, whilst behind the scenes we know that data is being written quickly, safely and that we can scale it as we continue to grow.

  9. Creating custom sounds for iPhone and Android mobile apps

    Leave a Comment

    With the new server monitoring mobile apps for iPhone and Android, we decided to fulfil one of the most common requests we had with our old apps – custom alert sounds. This is important because you want to ensure that you notice alerts coming in from your server and website monitoring alerts, and distinguish them from other push notifications on your phone.

    Both iPhone and Android allow you to specify custom sounds for push notifications so we hired a sound designer and composer to create some custom sounds just for our apps. The brief was to create a range of custom sounds for alerts – to be able to wake people up but still sound nice i.e a broad scope to leave the creativity up to the expert.

    We also asked them to document the process…enter Chris Rowan, the composer we worked with on this project:

    I’m Chris Rowan, freelance sound designer and composer. My specialties lie in creating original sounds for different types of media. For Server Density I developed alert sounds for the purpose of alerting customers when a server is malfunctioning. When creating the sounds for the app, my goal was to create a range of audible alerts that would contrast the usual tones and jingles associated with most apps. These contrasting alerts give an interesting range, helping users distinguish between Server Density and everyday mobile alerts. Three techniques where used making the server density application alert:

    1. MIDI
    2. Synthesis
    3. The manipulation of recorded sounds from natural sources

    Below is an analysis of three alternate sounds using the alternate techniques described above.

    Midi Instruments

    When using audio editing software, alert sounds can be developed through midi instruments. This is achieved through the manipulation of the instrument’s tone and body through modulation and alteration of the instrument’s parameters, such as flange or chorus.

    Midi instruments

    One of the sounds that utilised this technique is the manori stab (named after the MIDI instrument used).





    The original manori stab has a cascading element that I liked and thought would work well as an alert, but the length was too short and the attack to quick. To achieve the desired sound, pitch shifting and time stretching was required. These tools lowered the tone of the sound and also elongated it.





    Time Stretching can be extremely destructive so it is always best to use it sparingly. However, pitch shifting is a non destructive technique that can effectively heighten or lower the pitch of the sound. Pitch shifting is a better option and will use less memory to generate.

    Synthesis

    To create a truly original sound, digital synthesis gives you the means. Most sound editing software will have some sort of synth tool but for the sound ‘Delays on Delays’ I used an analogue synth.





    I chose to do this because having a physical medium to create can give you more accurate results making experimenting a lot faster and easier. The ‘Nord Rack X2′ Synth was used to create ‘Delays’, starting with a pure tone (Sine Wave), adjusting parameters and altering the attack, sustain, release and decay.

    Nord Rock Synth

    Synthesis can be used to create extremely complex sounds from the ground up and is used to make a lot of mobile device ringtones and alerts.

    Manipulation of recordings

    To contrast the synth heavy sounds described perviously, I recorded a range of natural sounds from wooden knocks to glass squeaks. After collecting the materials needed, I began recording a collection of sounds that would then be mixed, cut, and EQ’ed into desirable tones. I also utilised reverb to give the sounds more tail and body.

    EQ

    For the sound ‘Glass tones – Toned Down’ I recorded the sound of a finger running over the rim of a half full wine glass. This creates a looping tone that oscillates.

    Wine glass recording

    Once the sound was moved into Logic Pro, cutting was achieved using the sample editor. EQ, modulation and pitch shifting were used to alter the tone and make the sound brighter. Reverb was added to give a smooth tail and bigger body.





    When manipulating recordings it is important to apply effects gradually as plugins such as distortion and modulation will amplify unwanted artefacts previously unheard.

    The final choice

    The entire process created 14 sounds, most of which we have included in the app so the user can choose their favourite. Based on feedback we’ll see if we need any more created. After an internal vote we decided to make “space ring” the default, as a nice mixture of a futuristic sci-fi and alert tone.





    To see more of my work please check out my YouTube channel and LinkedIn.

    … and we’re back!

    So thanks to Chris Rowan for his fantastic work on our alert sounds and for documenting it for us to enjoy. If you’re interested in getting your hands on our new iPhone app, then it is available over here (If you haven’t got an account with us you’ll need to signup for your free trial)!

  10. What’s new in Server Density – Feb 2014

    Leave a Comment

    Each month we’ll round up all the feature changes and improvements we made this month to our server and website monitoring product, Server Density.

    New server overview

    Previously you had to configure a range of graphs for each of your devices, but we found that most people just set up a few standard graphs. Although the most common are defaulted, we have designed a new overview for each server which reveals all the key metrics without needing any configuration.

    Server overview

    This includes disk usage, CPU, load average, disk i/o and networking. You can view the graphs over the configured time range as well as the current, latest reported value. And the data auto refreshes when the agent posts in. The old graphs still remain under the “Metrics” tab so you can continue to add custom graphs for other metrics not in the default view.

    iPhone mobile app

    You can now get push notification alerts directly to your iPhone with our official, native server monitoring iPhone app available for free for all v2 accounts. It includes custom sounds too!

    iPhone server monitoring app - Server Density

    Advanced process monitoring alerting

    You can now configure a number of different process alerts:

    • Is a process running? Get alerted when processes crash or stop running e.g. is Apache running?
    • How many processes are running? Get alerted when processes go over or below a certain number e.g. are there at least 10 Apache worker processes?
    • Process CPU usage Get alerted when a specific process uses a certain amount of CPU resources e.g. is Apache using too much CPU?
    • Process memory usage
      Get alerted when a specific process uses a certain amount of memory e.g. is Apache using too much memory?

    Process monitoring

    New China monitoring node

    For service web checks you can now add our monitoring node in China, near Shanghai, to see if your site is loading from within the region.

    Advanced CPU monitoring alerting

    If you’re running the latest version of the monitoring agent for Linux, you will have the new “ALL” metric for CPUs, which gives you an average across all your CPU cores for things like idle, io wait, user, etc. You can now choose to have alerts triggered on specific CPU cores, if any of the cores match the alert config, or if the average across all cores matches the config. This is much more fine grained alerting than previously possible.

    CPU alert

    SPDY 3.1

    We’ve updated our load balancers to offer the latest SPDY 3.1 protocol which includes performance enhancements for multiplexing many HTTP requests. As per Cloudflare:

    A key advantage of SPDY is its ability to multiplex many HTTP request streams onto a single TCP connection. In the past, various hacks (such as domain sharding) have been used to get around the fact that only sequential, synchronous requests were possible with HTTP over TCP. SPDY changed all that.

    SPDY/3 introduced flow control so that SPDY clients (and servers) could control the amount of data they receive on a SPDY connection. SPDY/3.1 extended flow control to individual SPDY streams (each SPDY connection handles multiple simultaneous streams of data). Flow control is important because different clients (think of the differences in available memory in laptops, desktops and mobile phones) will have varying limitations on how much data they can receive at any one time.

    This means you should see better performance if you’re using the latest browser versions. Currently we only officially support Chrome, although Firefox does generally work and will be officially supported soon.