When would ever need more than 7 replica set members?

MongoDB supports the concept of replica sets which is like master/slave, but any member can become master automatically if the current master fails. It allows for automated, near instant failover. We’re using it live in production and each of our database shards has 4 members + 1 arbiter. MongoDB has a limit of 7 members max, but why would ever need that many? Why do we need 4 copies of our data?

This came up on the MongoDB mailing list and the discussion includes facts like even Google only has 3 or 4 copies of data. That’s true, but these are copies of data in multiple redundant locations. We use 2 locations (2 members per location) to provide geographic redundancy, but we also have 2 copies of the data in each location.

Why?

Each of our members stores 400GB of data. When a member goes down, it will be brought back online again at some point in the future and at that time, it needs to resync with the master from where it left off. If the downtime was only short, it can do a resync via the oplog but if the downtime was longer than the oplog last for, it will need to do a full resync – copying the entire database.

A full resync is a heavy operation and over a WAN (such as between data centres) it can take a very long time for 400GB of data – several days. Instead, we have a copy of the data already in the same data centre on a local network, so the resync can be completed much faster.

Further, we could shut down the other member in the same data centre and copy it to a new server. We use a private cloud so we can do that very easily and quickly. This means we have 2 recovery options. The resync reads from the master which has a performance hit, whereas copying would not.

If you have small data sets then this doesn’t matter, but for us it does. 7 is a good number because it means you can have 2 members across 3 locations + 1 arbiter. And if Google doesn’t need more than 3 locations, you probably don’t either!