UPS delivers at 11.22MB/s
We recently had one of our MongoDB database slaves go out of sync which requires a full resync from scratch. Because we have our slaves in a different data centre, it takes quite a lot of time for the data to transfer, indexes to be built and the slave to get back up to date again; 6 days in fact.
Due to the time it took, our oplog on the master was too small – it has to hold all activity on the master from when the sync starts to when it finishes so when the initial sync is completed, the slave can run through all the operations and get up to date again. We could have increased the size but it was already at 75GB and would require double that size. This is sufficient for the slave to be taken offline even for a several days (for maintenance for example) and then be able to catch back up again, but is not sufficient for a full resync.
150GB is too large so, we decided to have our hosting provider, Rackspace, plug a USB drive into the master, run a slave off the disk so it would be copying the data locally and then physically ship the disk to the secondary data centre. This was all set up and the USB drive was unplugged at 16:09 UTC on 12th Aug 2010 after spending a few days syncing the slave. Rackspace passed the drive to UPS who then took the drive from our primary data centre in Virginia to the secondary data centre in Texas.
The USB drive was plugged back into our server on 13th Aug 2010 at 18:07 UTC with a total elapsed time of 1 day, 1 hour, 58 minutes, or 93,480 seconds. After we’d copied all the data to the internal disks, the USB drive was shipped to me in the UK.
Using the inter-data-centre transfer time, we can figure out the data transfer rate:
500GB / 93480 = 0.005348737697903 GB/s or
5.477107402652672 MB/s (
If I use scp to copy a large file across between the two servers, then I get a slightly faster speed:
david@asriel ~: scp email@example.com:~/local/local.11 . local.11 100% 2047MB 6.8MB/s 05:01
However, whilst these figures are correct for the amount of data we transferred, the USB drive was actually 1TB and so the real data transfer rate is different:
1024GB / 93480 = 0.010954214805306GB/s or
11.217115960633343 MB/s (
It’s interesting to see these real world figures, although not entirely surprising. Pingdom wrote a short post about how FedEx is still faster than the internet back in 2007, particularly for large data sets. High speed connectivity is expensive and of course there’s always the speed of light limit. Hopefully that’ll be solved by quantum networks sometime in the future!
Enjoy this post? You may also like Devops London Demographics