Handling Memcached failover
CEO & Founder of Server Density.
Published on the 25th January, 2012.
Unlike a database with built in failover (master/slave model), you can usually only connect a client to a single Memcached server. If you specify multiple servers then these are used as part of the hashing to determine where the data gets stored, but there’s no concept of replication. This means if one Memcached node goes down, you lose the keys on that node. If you’re only connecting to a single node then you lose all Memcached.
The commercial product, Membase, handles this by providing replicated Memcached and failover functionality so if one node goes down, you can still access the other node(s) without any impact to the application. However, the clients are not built to handle this (it’s not standard Memcached functionality) so you still have to connect to a single node. You can’t use a pool of servers because Membase handles the distribution of data.
Instead, you can use the Moxi Memcached proxy. This allows your application servers to connect to what looks like a single Memcached host but Moxi handles sending the queries to the correct Membase (or Memcached) node. It also communicates with Membase to determine the health of a node for failover purposes.
We have recently deployed Moxi to elimiate Memcached as a single point of failure. Our web nodes now connect to one of several local Moxi instances (one for each Memcached bucket) which proxy the connections out to the cluster. If one of the Memcached cluster nodes fails, our application never needs to know because Moxi will silently handle the failover.
Alternatively, with Couchbase 1.8 (which is what Membase has been renamed to), you can use their client libraries to connect directly to your Couchbase instances with the failover support built into the libraries.