UPS delivers at 11.22MB/s

Written by David Mytton

We recently had one of our MongoDB database slaves go out of sync which requires a full resync from scratch. Because we have our slaves in a different data centre, it takes quite a lot of time for the data to transfer, indexes to be built and the slave to get back up to date again; 6 days in fact.

Due to the time it took, our oplog on the master was too small – it has to hold all activity on the master from when the sync starts to when it finishes so when the initial sync is completed, the slave can run through all the operations and get up to date again. We could have increased the size but it was already at 75GB and would require double that size. This is sufficient for the slave to be taken offline even for a several days (for maintenance for example) and then be able to catch back up again, but is not sufficient for a full resync.

150GB is too large so, we decided to have our hosting provider, Rackspace, plug a USB drive into the master, run a slave off the disk so it would be copying the data locally and then physically ship the disk to the secondary data centre. This was all set up and the USB drive was unplugged at 16:09 UTC on 12th Aug 2010 after spending a few days syncing the slave. Rackspace passed the drive to UPS who then took the drive from our primary data centre in Virginia to the secondary data centre in Texas.

The USB drive was plugged back into our server on 13th Aug 2010 at 18:07 UTC with a total elapsed time of 1 day, 1 hour, 58 minutes, or 93,480 seconds. After we’d copied all the data to the internal disks, the USB drive was shipped to me in the UK.

Using the inter-data-centre transfer time, we can figure out the data transfer rate:

500GB / 93480 = 0.005348737697903 GB/s or
5.477107402652672 MB/s (5.48MB/s rounded)

If I use scp to copy a large file across between the two servers, then I get a slightly faster speed:

david@asriel ~: scp david@internal.stelmaria.boxedice.net:~/local/local.11 .
local.11                                      100% 2047MB   6.8MB/s   05:01

However, whilst these figures are correct for the amount of data we transferred, the USB drive was actually 1TB and so the real data transfer rate is different:

1024GB / 93480 = 0.010954214805306GB/s or
11.217115960633343 MB/s (11.22MB/s rounded)

It’s interesting to see these real world figures, although not entirely surprising. Pingdom wrote a short post about how FedEx is still faster than the internet back in 2007, particularly for large data sets. High speed connectivity is expensive and of course there’s always the speed of light limit. Hopefully that’ll be solved by quantum networks sometime in the future!

Enjoy this post? You may also like Devops London Demographics

http://www.619cloud.com 619Cloud

Also, there is no charge for the bandwidth transfer using UPS, though you guys probably have an unmetered connection correct?
- http://www.serverdensity.com David Mytton
  
  It’s not unmetered but our transfer allocation is so high it’s effectively unlimited :)
Cabbey

This is an old truism. Back in the day the saying was “never underestimate a station wagon full of tapes barreling down the highway.” The grey beards I learned unix from used to say this was how they (in southern California) used to get code from the folks at Berkley so their unix machines could talk to everyone else’s unix machines over the phone. The idea of transferring that much code over the modems of the day was just reaching the point of not being a joke then.
giltotherescue

I’d love to see a blog post on what factors drove you to setup in a second data center, and what other challenges you’ve faced as a result.
- http://www.serverdensity.com David Mytton
  
  There is currently no single server durability in Mongo so you need to replicate to mitigate against hardware failure, power failures etc. Only way to isolate against all those is by using a geographically separate data centre.