redis-cluster 集群¶
一. Redis理论¶
尽管可以使用烧饼主从集群实现可用性保证，但是这种实现方式每个结点的数据都是全量复制，数据存放了存在着局限性。受限于内存的最小节点。因此考虑采用数据分片的方式来实现存储。这个就是redis-cluster
Redis cluster时redis的分布式解决方案。再3.0版本正式推出后，有效解决了redis分布式方面的需求；当遇到单机内存，并发，流量等瓶颈时，可以采用cluster架构方案来达到负载均衡的目的。
在3.0之前的版本要实现集群一般是借助哨兵sentinel工具来监控master节点的状态，如果master节点异常，则会主从切换。将某一台slave作为master。少斌的配置略微复杂，并且性能和高可用性等各方面表现一般。特别时主从切换的瞬间存在访问瞬断的情况。而且哨兵模式只有一个主节点对外提供服务。没办法持续很高的并发。且单个主节点内存也不宜设置过大，否则会导致持久化文件过大，影响数据恢复或主从同步的效率。
二. Redis数据分区¶
Redis cluster采用虚拟槽分区，所有的键值根据哈希函数映射到0~16383整数槽中。每一个节点负责维护一部分槽一级槽所映射的键值数据。
Redis群集中每一个节点都负责哈希槽的子集，例如您可能拥有一个包含三个节点的群集。其中：
节点a包含从0到5460的散列槽
节点b包含从5461到10922的散列槽
节点c包含从10923到16383的散列槽
1. Redis Cluster主从模型¶
如果只具有节点A,B,C实例的集群中，如果节点B发生故障，则群集无法继续。因为我们不在能够在5461-10922范围内提供服务哈希位置的方法
然而，当创建集群时(或稍后)，我们像每个主节点添加一个从节点。一边最终集群由作为主节点的A,B,C和作为从节点的A1,B1,C1组成。如果节点B出现故障，系统就能继续允许。
节点B1复制节点B。当B失去联系时，集群将节点B1升级为新的主节点，并继续正常运行。
但请注意，如果节点B和B1同时发生故障时，Redis Cluster将无法继续运行
Redis Cluster一般由多个节点组成，节点数量至少为六个才能保证组成完整的高可用集群。其中三个为主节点，三个为从节点。三个主节点会分配槽，处理客户端的请求。而从节点可用在主节点故障后顶替主节点
2. Redis-cluster搭建¶
Redis集群至少需要三个节点，因为投票容错机制要求超过半数节点认为某个节点挂了该节点才是挂了，所以两个节点无法构成集群
要保证集群的高可用，需要每个节点都有从节点，也就是备份节点。所以Redis集群至少需要六台服务器
我们这里搭建三个master节点，并且给每个master再搭建一个slave节点，三主三从
3. 实验环境¶
| 主机 | IP | 角色 | 端口 |
|---|---|---|---|
| 1 | 192.168.100.99 | Master | 6379 |
| 1 | 192.168.100.99 | Slave | 6380 |
| 2 | 192.168.100.100 | Master | 6379 |
| 2 | 192.168.100.100 | Slave | 6380 |
| 3 | 192.168.100.101 | Master | 6379 |
| 3 | 192.168.100.101 | Slave | 6380 |
4. 安装redis¶
5. 1设置¶
a. 修改hosts
b. 创建cluster工作目录
[root@1 ~]# vi /etc/hosts
[root@1 ~]# mkdir -p /etc/redis/cluster/{6379,6380}
[root@1 ~]# ls /etc/redis/cluster
6379 6380
c. 创建数据存储目录
[root@1 ~]# mkdir -p /data/redis/data/{6379,6380}
[root@1 ~]# mkdir /var/log/redis
[root@1 ~]# ll /data/redis/data
总用量 0
drwxr-xr-x 2 root root 6 7月 9 17:53 6379
drwxr-xr-x 2 root root 6 7月 9 17:53 6380
d. 生成配置文件
e. 修改6379的配置文件
daemonize yes //redis后台运行
bind 192.168.100.99 //监听地址
pidfile /var/run/redis_6379.pid //pidfile文件对应端口
port 6379 //监听的端口
dir "/data/redis/data/6379" //配置redis rdb数据保存位置
logfile "/var/log/redis/redis_6379.log"
protected-mode no //关闭保护模式，允许远程连接，将yes改成no
cluster-enabled yes //开启集群，把注释#去掉
cluster-config-file nodes_6379.conf //集群的配置，配置文件首次启动时自动生成
cluster-node-timeout 15000 //请求超时15s
appendonly yes //aof日志开启，有需求就开启，会每次写操作都记录一条日志
主要参数说明
cluster-enabled
cluster-config-file
luster-node-timeout
f. 修改6380的配置文件
[root@1 ~]# cp /etc/redis/cluster/6379 /etc/redis/cluster/6380
[root@1 ~]# sed -i 's@6379@6380@g' /etc/redis/cluster/6380/redis.conf
g. 启动redis服务
h. 查看端口占用情况
[root@1 ~]# ss -tuanp|grep 63
tcp LISTEN 0 511 192.168.100.99:6380 0.0.0.0:* users:(("redis-server",pid=3373,fd=9))
tcp LISTEN 0 511 192.168.100.99:6379 0.0.0.0:* users:(("redis-server",pid=3366,fd=8))
tcp LISTEN 0 511 192.168.100.99:16380 0.0.0.0:* users:(("redis-server",pid=3373,fd=10))
[root@1 ~]# ps -elf|grep redis
5 S root 3366 1 0 80 0 - 32682 ep_pol 10:23 ? 00:00:00 redis-server 192.168.100.99:6379
5 S root 3373 1 0 80 0 - 34090 ep_pol 10:23 ? 00:00:00 redis-server 192.168.100.99:6380 [cluster]
0 S root 3388 3298 0 80 0 - 1663 pipe_r 10:23 pts/0 00:00:00 grep --color=auto redis
6. 2,3设置¶
a. 修改hosts
b. 修改6379配置文件
cluster-enabled yes
bind 192.168.100.100
port 6379
pidfile /var/run/redis_6379.pid
logfile /var/log/redis/redis_6379.log dir
/var/lib/redis/6379
cluster-node-timeout 5000
c. 修改6380配置文件
[root@2 ~]# cp -a /etc/redis/redis-6379.conf /etc/redis/redis-6380.conf
[root@2 ~]# sed -i 's@6379@6380@g' /etc/redis/redis-6380.conf
d. 把改好的配置文件复制到3上
[root@2 ~]# scp /etc/redis/redis-6379.conf root@192.168.100.101:/etc/redis/
redis-6379.conf 100% 120KB 32.2MB/s 00:00
[root@2 ~]# scp /etc/redis/redis-6380.conf root@192.168.100.101:/etc/redis/
redis-6380.conf 100% 120KB 41.2MB/s 00:00
[root@3 ~]# sed -i 's@192.168.100.100@192.168.100.101@' /etc/redis/redis-6380.conf
[root@3 ~]# sed -i 's@192.168.100.100@192.168.100.101@' /etc/redis/redis-6379.conf
e. 分别创建数据目录文件
d. 启动服务
[root@2 ~]# redis-server /etc/redis/redis-6379.conf
redis-server /etc/redis/redis-6380.conf
[root@2 ~]# ss -tuanp|grep 63
tcp LISTEN 0 511 192.168.100.100:6379 0.0.0.0:* users:(("redis-server",pid=3760,fd=18))
tcp LISTEN 0 511 192.168.100.100:6380 0.0.0.0:* users:(("redis-server",pid=3763,fd=18))
[root@2 ~]# ps -elf|grep redis
5 S root 3760 1 0 80 0 - 44106 ep_pol 10:20 ? 00:00:00 redis-server 192.168.100.100:6379
5 S root 3763 1 0 80 0 - 44106 ep_pol 10:20 ? 00:00:00 redis-server 192.168.100.100:6380
0 S root 3781 1961 0 80 0 - 1663 pipe_r 10:20 pts/0 00:00:00 grep --color=auto redis
[root@3 ~]# redis-server /etc/redis/redis-6379.conf
redis-server /etc/redis/redis-6380.conf
[root@3 ~]# ss -tuanp|grep 63
tcp LISTEN 0 511 192.168.100.101:6379 0.0.0.0:* users:(("redis-server",pid=2953,fd=18))
tcp LISTEN 0 511 192.168.100.101:6380 0.0.0.0:* users:(("redis-server",pid=2955,fd=18))
[root@3 ~]# ps -elf|grep redis
5 S root 2953 1 0 80 0 - 44106 ep_pol 10:18 ? 00:00:00 redis-server 192.168.100.101:6379
5 S root 2955 1 0 80 0 - 44106 ep_pol 10:18 ? 00:00:00 redis-server 192.168.100.101:6380
0 S root 2977 1871 0 80 0 - 1663 pipe_r 10:19 pts/0 00:00:00 grep --color=auto redis
彻底杀死redis进程后重启命令
7. 创建cluster集群¶
Ⅰ. 添加集群
redis-cli --cluster create 192.168.100.99:6379 192.168.100.99:6380 \
192.168.100.100:6379 192.168.100.100:6380 \
192.168.100.101:6379 192.168.100.101:6380 --cluster-replicas 1
错误：
[root@1 ~]# redis-cli --cluster create 192.168.100.99:6379 192.168.100.99:6380 \
192.168.100.100:6379 192.168.100.100:6380 \
192.168.100.101:6379 192.168.100.101:6380 --cluster-replicas 1
[ERR] Node 192.168.100.99:6379 DENIED Redis is running in protected mode because protected mode is enabled and no password is set for the default user. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Set up an authentication password for the default user. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.
原因：1的保护模式没有关闭
把protected-mode yes改为no
[root@1 ~]# sed -i 's@protected-mode yes@protected-mode no@' /etc/redis/cluster/6379/redis.conf
sed -i 's@protected-mode yes@protected-mode no@' /etc/redis/cluster/6380/redis.conf
[root@1 ~]# systemctl stop redis 2>/dev/null
pkill -9 redis-server
ss -tuanp | grep redis
[root@1 ~]# redis-server /etc/redis/cluster/6379/redis.conf
redis-server /etc/redis/cluster/6380/redis.conf
8. 启动集群¶
[root@1 ~]# redis-cli --cluster create 192.168.100.99:6379 192.168.100.99:6380 \
192.168.100.100:6379 192.168.100.100:6380 \
192.168.100.101:6379 192.168.100.101:6380 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 192.168.100.100:6380 to 192.168.100.99:6379
Adding replica 192.168.100.101:6380 to 192.168.100.100:6379
Adding replica 192.168.100.99:6380 to 192.168.100.101:6379
M: 226b194b5bf94da06fd94d13089c06a2cda0eb7c 192.168.100.99:6379
slots:[0-5460] (5461 slots) master
S: 52d3b1bc73beb3f9e73771cae5b3f8c51ec68629 192.168.100.99:6380
replicates da8cd5bb0fc05960fd14198cb247a41a825fcd9d
M: f1e4fcd777630b29aa4f207bcd472c6c93075dd9 192.168.100.100:6379
slots:[5461-10922] (5462 slots) master
S: 62700260ae2f77e91bfd0b667e329d7b6f3e7de0 192.168.100.100:6380
replicates 226b194b5bf94da06fd94d13089c06a2cda0eb7c
M: da8cd5bb0fc05960fd14198cb247a41a825fcd9d 192.168.100.101:6379
slots:[10923-16383] (5461 slots) master
S: 64b01ca46b601de5d689bc558279b3e5207cd1ba 192.168.100.101:6380
replicates f1e4fcd777630b29aa4f207bcd472c6c93075dd9
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
>>> Performing Cluster Check (using node 192.168.100.99:6379)
M: 226b194b5bf94da06fd94d13089c06a2cda0eb7c 192.168.100.99:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 62700260ae2f77e91bfd0b667e329d7b6f3e7de0 192.168.100.100:6380
slots: (0 slots) slave
replicates 226b194b5bf94da06fd94d13089c06a2cda0eb7c
M: da8cd5bb0fc05960fd14198cb247a41a825fcd9d 192.168.100.101:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 64b01ca46b601de5d689bc558279b3e5207cd1ba 192.168.100.101:6380
slots: (0 slots) slave
replicates f1e4fcd777630b29aa4f207bcd472c6c93075dd9
S: 52d3b1bc73beb3f9e73771cae5b3f8c51ec68629 192.168.100.99:6380
slots: (0 slots) slave
replicates da8cd5bb0fc05960fd14198cb247a41a825fcd9d
M: f1e4fcd777630b29aa4f207bcd472c6c93075dd9 192.168.100.100:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
9. 查看集群信息¶
查看节点信息
[root@1 ~]# redis-cli -h 192.168.100.99 -p 6379 -c
192.168.100.99:6379> cluster nodes
[root@1 ~]# redis-cli -h 192.168.100.99 -p 6379 -c
192.168.100.99:6379> cluster nodes
52d3b1bc73beb3f9e73771cae5b3f8c51ec68629 192.168.100.99:6380@16380 slave da8cd5bb0fc05960fd14198cb247a41a825fcd9d 0 1783652662244 5 connected
64b01ca46b601de5d689bc558279b3e5207cd1ba 192.168.100.101:6380@16380 slave f1e4fcd777630b29aa4f207bcd472c6c93075dd9 0 1783652663252 3 connected
226b194b5bf94da06fd94d13089c06a2cda0eb7c 192.168.100.99:6379@16379 myself,master - 0 0 1 connected 0-5460
da8cd5bb0fc05960fd14198cb247a41a825fcd9d 192.168.100.101:6379@16379 master - 0 1783652662000 5 connected 10923-16383
f1e4fcd777630b29aa4f207bcd472c6c93075dd9 192.168.100.100:6379@16379 master - 0 1783652662747 3 connected 5461-10922
62700260ae2f77e91bfd0b667e329d7b6f3e7de0 192.168.100.100:6380@16380 slave 226b194b5bf94da06fd94d13089c06a2cda0eb7c 0 1783652661236 1 connected
写入数据
查看集群信息
192.168.100.101:6379> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:5
cluster_stats_messages_ping_sent:261
cluster_stats_messages_pong_sent:260
cluster_stats_messages_meet_sent:1
cluster_stats_messages_sent:522
cluster_stats_messages_ping_received:260
cluster_stats_messages_pong_received:264
cluster_stats_messages_received:524
total_cluster_links_buffer_limit_exceeded:0
cluster_slot_migration_active_tasks:0
cluster_slot_migration_active_trim_running:0
cluster_slot_migration_active_trim_current_job_keys:0
cluster_slot_migration_active_trim_current_job_trimmed:0
cluster_slot_migration_stats_active_trim_started:0
cluster_slot_migration_stats_active_trim_completed:0
cluster_slot_migration_stats_active_trim_cancelled:0
查看集群完整性
[root@3 ~]# redis-cli --cluster check 192.168.100.101:6379
192.168.100.101:6379 (da8cd5bb...) -> 1 keys | 5461 slots | 1 slaves.
192.168.100.99:6379 (226b194b...) -> 0 keys | 5461 slots | 1 slaves.
192.168.100.100:6379 (f1e4fcd7...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 1 keys in 3 masters.
>>> Performing Cluster Check (using node 192.168.100.101:6379)
M: da8cd5bb0fc05960fd14198cb247a41a825fcd9d 192.168.100.101:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: 226b194b5bf94da06fd94d13089c06a2cda0eb7c 192.168.100.99:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 52d3b1bc73beb3f9e73771cae5b3f8c51ec68629 192.168.100.99:6380
slots: (0 slots) slave
replicates da8cd5bb0fc05960fd14198cb247a41a825fcd9d
M: f1e4fcd777630b29aa4f207bcd472c6c93075dd9 192.168.100.100:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 64b01ca46b601de5d689bc558279b3e5207cd1ba 192.168.100.101:6380
slots: (0 slots) slave
replicates f1e4fcd777630b29aa4f207bcd472c6c93075dd9
S: 62700260ae2f77e91bfd0b667e329d7b6f3e7de0 192.168.100.100:6380
slots: (0 slots) slave
replicates 226b194b5bf94da06fd94d13089c06a2cda0eb7c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
...
10. 故障切换¶
模拟主redis服务192.168.100.101：6379故障
[root@3 ~]# ss -tuanp | grep 6379 | grep LISTEN
tcp LISTEN 0 511 192.168.100.101:6379 0.0.0.0:* users:(("redis-server",pid=5412,fd=19))
tcp LISTEN 0 511 192.168.100.101:16379 0.0.0.0:* users:(("redis-server",pid=5412,fd=20))
[root@3 ~]# kill -9 5412
检查集群完整性
[root@1 ~]# redis-cli --cluster check 192.168.100.99:6379
192.168.100.99:6379 (226b194b...) -> 0 keys | 5461 slots | 1 slaves.
192.168.100.100:6379 (f1e4fcd7...) -> 0 keys | 5462 slots | 1 slaves.
192.168.100.101:6379 (da8cd5bb...) -> 1 keys | 5461 slots | 1 slaves.
[OK] 1 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.100.99:6379)
M: 226b194b5bf94da06fd94d13089c06a2cda0eb7c 192.168.100.99:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: f1e4fcd777630b29aa4f207bcd472c6c93075dd9 192.168.100.100:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 52d3b1bc73beb3f9e73771cae5b3f8c51ec68629 192.168.100.99:6380
slots: (0 slots) slave
replicates da8cd5bb0fc05960fd14198cb247a41a825fcd9d
S: 62700260ae2f77e91bfd0b667e329d7b6f3e7de0 192.168.100.100:6380
slots: (0 slots) slave
replicates 226b194b5bf94da06fd94d13089c06a2cda0eb7c
M: da8cd5bb0fc05960fd14198cb247a41a825fcd9d 192.168.100.101:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 64b01ca46b601de5d689bc558279b3e5207cd1ba 192.168.100.101:6380
slots: (0 slots) slave
replicates f1e4fcd777630b29aa4f207bcd472c6c93075dd9
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[root@1 ~]# sleep 20 && redis-cli --cluster check 192.168.100.99:6379
Could not connect to Redis at 192.168.100.101:6379: Connection refused
*** WARNING: 192.168.100.99:6380 claims to be slave of unknown node ID da8cd5bb0fc05960fd14198cb247a41a825fcd9d.
192.168.100.99:6379 (226b194b...) -> 0 keys | 5461 slots | 1 slaves.
192.168.100.100:6379 (f1e4fcd7...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 2 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.100.99:6379)
M: 226b194b5bf94da06fd94d13089c06a2cda0eb7c 192.168.100.99:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: f1e4fcd777630b29aa4f207bcd472c6c93075dd9 192.168.100.100:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 52d3b1bc73beb3f9e73771cae5b3f8c51ec68629 192.168.100.99:6380
slots: (0 slots) slave
replicates da8cd5bb0fc05960fd14198cb247a41a825fcd9d
S: 62700260ae2f77e91bfd0b667e329d7b6f3e7de0 192.168.100.100:6380
slots: (0 slots) slave
replicates 226b194b5bf94da06fd94d13089c06a2cda0eb7c
S: 64b01ca46b601de5d689bc558279b3e5207cd1ba 192.168.100.101:6380
slots: (0 slots) slave
replicates f1e4fcd777630b29aa4f207bcd472c6c93075dd9
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[ERR] Not all 16384 slots are covered by nodes.
[root@1 ~]# redis-cli --cluster check 192.168.100.99:6379
Could not connect to Redis at 192.168.100.99:6380: Connection refused
192.168.100.99:6379 (226b194b...) -> 0 keys | 5461 slots | 1 slaves.
192.168.100.101:6379 (da8cd5bb...) -> 0 keys | 5461 slots | 0 slaves.
192.168.100.100:6379 (f1e4fcd7...) -> 0 keys | 5462 slots | 1 slaves.
[OK] 0 keys in 3 masters.
0.00 keys per slot on average.
>>> Performing Cluster Check (using node 192.168.100.99:6379)
M: 226b194b5bf94da06fd94d13089c06a2cda0eb7c 192.168.100.99:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: 64b01ca46b601de5d689bc558279b3e5207cd1ba 192.168.100.101:6380
slots: (0 slots) slave
replicates f1e4fcd777630b29aa4f207bcd472c6c93075dd9
M: da8cd5bb0fc05960fd14198cb247a41a825fcd9d 192.168.100.101:6379
slots:[10923-16383] (5461 slots) master
M: f1e4fcd777630b29aa4f207bcd472c6c93075dd9 192.168.100.100:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 62700260ae2f77e91bfd0b667e329d7b6f3e7de0 192.168.100.100:6380
slots: (0 slots) slave
replicates 226b194b5bf94da06fd94d13089c06a2cda0eb7c
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
再次启动3
[root@3 ~]# redis-server /etc/redis/redis-6379.conf
[root@3 ~]# ss -tuanp | grep 6379 | grep LISTEN
tcp LISTEN 0 511 192.168.100.101:6379 0.0.0.0:* users:(("redis-server",pid=5412,fd=19))
tcp LISTEN 0 511 192.168.100.101:16379 0.0.0.0:* users:(("redis-server",pid=5412,fd=20))
查看情况
systemctl stop redis 2>/dev/null pkill -9 redis-server redis-server /etc/redis/cluster/6379/redis.conf redis-server /etc/redis/cluster/6380/redis.conf
systemctl stop redis 2>/dev/null pkill -9 redis-server redis-server /etc/redis/redis-6379.conf redis-server /etc/redis/redis-6380.conf