This article describes how to configure active-passive high availability(HA) for a group of Redis instances. I avoid using cluster because Redis Cluster is the mechanism to distribute shards to multiple Redis instances . Active-passive means there is only one Redis instance, the master, serving requests at all time. Data is replicated from the master to the rest of the Redis instances, the slaves. One of the slaves would replace the master when the master is down. Therefore HA essentially needs to take care of two major issues, data availability and service availability. Redis replication provides data availability. Redis Sentinel ensures service availability. Redis Sentinel monitors Redis instances. For example, consider a group of three Redis instances, A, B, and C, with A as the initial master. The following figure illustrate how Redis Sentinel monitors Redis instances.
When the master is down, Redis Sentinel automatically elects a new master and failover to the the new master as shown in the following figure.
The following sections shows the steps of configuring Rdis HA. The example assumes:
- CentOS 6.5.
Redis instances are installed with rpm packages, version > 2.6.
Host IP and ports.
IP Redis Port Redis Sentinel Port A 10.211.55.10 6379 26379 B 10.211.55.11 6379 26379 C 10.211.55.12 6379 26379
Data: Replication
Replication is asynchronous because of performance concerns. It means that Redis HA does not guarantee strong data consistency. Data loss might happen. You can find the detail design on redis.io.
"However, because Redis uses asynchronous replication it is not possible to ensure the slave actually received a given write, so there is always a window for data loss."
To enable replication, add directive slaveof to the configuration files of instance B and C. Edit /etc/redis.conf.
# slaveof master.ip 6379 slaveof 10.211.55.10 6379 |
Then restart Redis.
/etc/init.d/redis restart |
DO NOT enable replication in master node. Redis Sentinel would treat the master as failed node and keep trying failover to the slaves.
To verify that replication is working properly, login to the master and write some data.
[mymaster]# redis-cli 10.211.55.10:6379> set hello world |
Now login to one of the slave.
[slave]# redis-cli 10.211.55.11:6379> get hello "world" |
Service Availability: Sentinel
The next step is to configure Redis Sentinel to monitor Redis instances. In the following example Redis Sentinel runs on the same node as Redis instance although Redis Sentinel can run anywhere you want.
# sentinel monitor <master-name> <ip> <redis-port> <quorum> sentinel monitor mymaster 10.211.55.10 6379 2 sentinel down-after-milliseconds mymaster 60000 sentinel failover-timeout mymaster 180000 sentinel parallel-syncs mymaster 1 |
Edit /etc/redis-sentinel.conf of each redis-sentiel and add the following configurations.
Line 2 is the most important configuration. It tells Redis Sentinel that the master is 10.211.55.10, the master's name is "mymaster", and start failover if more than two Redis Sentinel has detected the master failed. You can find explanations on redis.io.
Then start all Redis Sentinel on each node.
/etc/init.d/redis-sentinel start |
Client
If you have existing applications use Redis, you will need to modify two parts . First make sure your client library support Redis Sentinel. Second instead of connecting to a single Redis instance, use Sentinel API to discover the master. Then operate on the master. The following code segments are modified from redis-py.
from redis.sentinel import Sentinel
sentinel = Sentinel([('10.211.55.10', 26379)], [('10.211.55.11', 26379)], [('10.211.55.12', 26379)], socket_timeout=0.1)
sentinel.discover_master('mymaster')
#('10.211.55.10', 6379)
sentinel.discover_slaves('mymaster')
#[('10.211.55.11', 6379), ('10.211.55.12', 6379)]
master = sentinel.master_for('mymaster', socket_timeout=0.1)
master.set('foo', 'bar') |
References
- Redis replication, http://redis.io/topics/replication
- Redis sentinel, http://redis.io/topics/sentinel