Redis Cluster集群新增、删除节点以及重新分配hash slot哈希槽

NoSQL笔记 2020年05月27日

本篇笔记记录了Redis Cluster集群(扩容)新增节点、(缩容)删除节点的方法,以及如何在(扩容)新增和(缩容)删除节点时对hash slot(哈希槽)进行重新分配

Redis集群搭建请参考 Redis Cluster集群搭建

当前Redis集群节点

192.168.75.240:6379	192.168.75.240:6380
192.168.75.241:6379	192.168.75.241:6380
192.168.75.242:6379	192.168.75.242:6380

什么是hash slot(哈希槽)

援引作者说法:

Redis Cluster没有使用一致哈希,而是使用一种不同形式的分片,每个key从概念上讲都是我们称为hash slot的一部分。
Redis Cluster中有16384个hash slot,要计算某个key所属的hash slot,我们只需对key的CRC16值和16384取模。

简洁的说,一致性哈希计算key分布使用Time33(也叫DJBX33A)算法,Redis Cluster计算key分布使用CRC16结果和16384取模;更简洁的说,一致性哈希将key分布在2的32次方个节点上,Redis Cluster将key分布在16384个哈希槽上,Redis Cluster集群中每个主节点分布一个哈希槽的话,Redis Cluster集群最多容纳16384个主节点,当然,实际应用中不可能有那么多主节点。

查看当前集群节点和哈希槽分布

登录任一节点

redis-cli -c -p 6379

-c表示集群
执行命令

cluster nodes

集群节点信息如下

127.0.0.1:6379> cluster nodes
c36e8e3175f53cccdf8640424192593c230ebd89 192.168.75.240:6380@16380 slave afad77c65770a962e53b89b275578a5ef382f039 0 1590295653703 5 connected
b16f676efe8d7998bfe56386434895ced77e79e7 192.168.75.240:6379@16379 myself,slave 726b7ba73ff0e4849a13fe1612ebc952d1c8786c 0 1590295652000 1 connected
afad77c65770a962e53b89b275578a5ef382f039 192.168.75.242:6379@16379 master - 0 1590295656000 5 connected 10923-16383
19f46bcef63286be0ebbcfd2acb9beae31f471d4 192.168.75.241:6379@16379 master - 0 1590295654745 3 connected 5461-10922
096e85ce032914c9c0f3107b6d3c49e612a57ab7 192.168.75.242:6380@16380 slave 19f46bcef63286be0ebbcfd2acb9beae31f471d4 0 1590295655000 6 connected
726b7ba73ff0e4849a13fe1612ebc952d1c8786c 192.168.75.241:6380@16380 master - 0 1590295656871 7 connected 0-5460
  • 第一列为节点ID
  • 第二列为节点IP:端口号@集群管理端口号
  • 第三列为主/从节点,myself表示当前执行命令的节点
  • 第四列为父节点ID,主节点的父节点ID为0
  • 主节点的最后一列为哈希槽分布区间,从节点同步主节点数据,槽位依据主节点槽位

设置key/value

127.0.0.1:6379> set key1 value1
-> Redirected to slot [9189] located at 192.168.75.241:6379
OK
192.168.75.241:6379> set key2 value2
-> Redirected to slot [4998] located at 192.168.75.241:6380
OK
192.168.75.241:6380> set key3 value3
OK
192.168.75.241:6380> set key4 value4
-> Redirected to slot [13120] located at 192.168.75.242:6379
OK
192.168.75.242:6379> set key5 value5
-> Redirected to slot [9057] located at 192.168.75.241:6379
OK
192.168.75.241:6379> set key6 value6
-> Redirected to slot [4866] located at 192.168.75.241:6380
OK
192.168.75.241:6380> set key7 value7
OK
192.168.75.241:6380> set key8 value8
-> Redirected to slot [13004] located at 192.168.75.242:6379
OK
192.168.75.242:6379> set key9 value9
-> Redirected to slot [8941] located at 192.168.75.241:6379
OK
192.168.75.241:6379> set key10 value10
OK

我设置了10个key/value,结果显示了key对应的哈希槽和redis节点IP:端口号信息

新增节点并分配哈希槽

任意服务器创建主节点和从节点,可以不在一台服务器上,为省事我选择在192.168.75.240上操作
创建Redis工作目录

mkdir -p /usr/local/redis/6381
mkdir -p /usr/local/redis/6382

复制安装包下配置文件到安装目录

cp /usr/local/src/redis-5.0.5/redis.conf /usr/local/redis/redis_6381.conf
cp /usr/local/src/redis-5.0.5/redis.conf /usr/local/redis/redis_6382.conf

修改配置文件
redis_6381.conf

bind 0.0.0.0
protected-mode no
port 6381
daemonize yes
pidfile /var/run/redis_6381.pid
logfile "/var/log/redis/6381.log"
dir /usr/local/redis/6381
cluster-enabled yes
cluster-config-file nodes-6381.conf
cluster-node-timeout 15000

redis_6382.conf

bind 0.0.0.0
protected-mode no
port 6382
daemonize yes
pidfile /var/run/redis_6382.pid
logfile "/var/log/redis/6382.log"
dir /usr/local/redis/6382
cluster-enabled yes
cluster-config-file nodes-6382.conf
cluster-node-timeout 15000

启动Redis

redis-server /usr/local/redis/redis_6381.conf
redis-server /usr/local/redis/redis_6382.conf

将两个节点加入集群
添加192.168.75.240:6381到192.168.75.240:6379所属的集群中作为master

redis-cli --cluster add-node 192.168.75.240:6381 192.168.75.240:6379

查看集群节点信息

redis-cli -c -p 6379 cluster nodes
c36e8e3175f53cccdf8640424192593c230ebd89 192.168.75.240:6380@16380 slave afad77c65770a962e53b89b275578a5ef382f039 0 1590297292519 5 connected
b16f676efe8d7998bfe56386434895ced77e79e7 192.168.75.240:6379@16379 myself,slave 726b7ba73ff0e4849a13fe1612ebc952d1c8786c 0 1590297291000 1 connected
afad77c65770a962e53b89b275578a5ef382f039 192.168.75.242:6379@16379 master - 0 1590297293564 5 connected 10923-16383
19f46bcef63286be0ebbcfd2acb9beae31f471d4 192.168.75.241:6379@16379 master - 0 1590297293000 3 connected 5461-10922
096e85ce032914c9c0f3107b6d3c49e612a57ab7 192.168.75.242:6380@16380 slave 19f46bcef63286be0ebbcfd2acb9beae31f471d4 0 1590297294599 6 connected
726b7ba73ff0e4849a13fe1612ebc952d1c8786c 192.168.75.241:6380@16380 master - 0 1590297292000 7 connected 0-5460
b7352e36c147dcba86cdc1103a066f5e95606eac 192.168.75.240:6381@16381 master - 0 1590297291498 0 connected

192.168.75.240:6381的ID为b7352e36c147dcba86cdc1103a066f5e95606eac
添加192.168.75.240:6382为192.168.75.240:6381的从节点到192.168.75.240:6379所属的集群

redis-cli --cluster add-node 192.168.75.240:6382 192.168.75.240:6379 --cluster-slave --cluster-master-id b7352e36c147dcba86cdc1103a066f5e95606eac

查看集群节点信息

redis-cli -c -p 6379 cluster nodes
069f4ff213f5ad8f2bd59424eb25e1bf7e3eed12 192.168.75.240:6382@16382 slave b7352e36c147dcba86cdc1103a066f5e95606eac 0 1590297530000 0 connected
c36e8e3175f53cccdf8640424192593c230ebd89 192.168.75.240:6380@16380 slave afad77c65770a962e53b89b275578a5ef382f039 0 1590297528000 5 connected
b16f676efe8d7998bfe56386434895ced77e79e7 192.168.75.240:6379@16379 myself,slave 726b7ba73ff0e4849a13fe1612ebc952d1c8786c 0 1590297524000 1 connected
afad77c65770a962e53b89b275578a5ef382f039 192.168.75.242:6379@16379 master - 0 1590297531469 5 connected 10923-16383
19f46bcef63286be0ebbcfd2acb9beae31f471d4 192.168.75.241:6379@16379 master - 0 1590297528000 3 connected 5461-10922
096e85ce032914c9c0f3107b6d3c49e612a57ab7 192.168.75.242:6380@16380 slave 19f46bcef63286be0ebbcfd2acb9beae31f471d4 0 1590297529349 6 connected
726b7ba73ff0e4849a13fe1612ebc952d1c8786c 192.168.75.241:6380@16380 master - 0 1590297530402 7 connected 0-5460
b7352e36c147dcba86cdc1103a066f5e95606eac 192.168.75.240:6381@16381 master - 0 1590297528319 0 connected

我们看到,192.168.75.240:6382以从节点身份加入到了集群,但新增的主节点192.168.75.240:6381还没有分配哈希槽

重新分配哈希槽

自动平衡集群中的各节点哈希槽数

redis-cli --cluster rebalance --cluster-use-empty-masters 192.168.75.240:6379

执行过程

>>> Performing Cluster Check (using node 192.168.75.240:6379)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 4 nodes. Total weight = 4.00
Moving 1366 slots from 192.168.75.241:6379 to 192.168.75.240:6381
###########################......
Moving 1365 slots from 192.168.75.241:6380 to 192.168.75.240:6381
###########################......
Moving 1365 slots from 192.168.75.242:6379 to 192.168.75.240:6381
###########################......

我们看到,系统自动从当前各主节点中拿出一定数量的哈希槽,转义至新的主节点中,以达到哈希槽平衡,Moving 1365 slots中的1365为移动的哈希槽数量
查看集群节点信息

redis-cli -c -p 6379 cluster nodes
069f4ff213f5ad8f2bd59424eb25e1bf7e3eed12 192.168.75.240:6382@16382 slave b7352e36c147dcba86cdc1103a066f5e95606eac 0 1590298680718 9 connected
c36e8e3175f53cccdf8640424192593c230ebd89 192.168.75.240:6380@16380 slave afad77c65770a962e53b89b275578a5ef382f039 0 1590298678000 5 connected
b16f676efe8d7998bfe56386434895ced77e79e7 192.168.75.240:6379@16379 myself,slave 726b7ba73ff0e4849a13fe1612ebc952d1c8786c 0 1590298681000 1 connected
afad77c65770a962e53b89b275578a5ef382f039 192.168.75.242:6379@16379 master - 0 1590298676528 5 connected 12288-16383
19f46bcef63286be0ebbcfd2acb9beae31f471d4 192.168.75.241:6379@16379 master - 0 1590298681000 3 connected 6827-10922
096e85ce032914c9c0f3107b6d3c49e612a57ab7 192.168.75.242:6380@16380 slave 19f46bcef63286be0ebbcfd2acb9beae31f471d4 0 1590298679000 6 connected
726b7ba73ff0e4849a13fe1612ebc952d1c8786c 192.168.75.241:6380@16380 master - 0 1590298680000 7 connected 1365-5460
b7352e36c147dcba86cdc1103a066f5e95606eac 192.168.75.240:6381@16381 master - 0 1590298681766 9 connected 0-1364 5461-6826 10923-12287

我们看到,主节点192.168.75.240:6381已经分配了哈希槽
那么我们刚才设置的key/value还在吗,我们测试一下

[root@localhost ~]# redis-cli -c -p 6379
127.0.0.1:6379> get key1
-> Redirected to slot [9189] located at 192.168.75.241:6379
"value1"
192.168.75.241:6379> get key2
-> Redirected to slot [4998] located at 192.168.75.241:6380
"value2"
192.168.75.241:6380> get key3
-> Redirected to slot [935] located at 192.168.75.240:6381
"value3"
192.168.75.240:6381> get key4
-> Redirected to slot [13120] located at 192.168.75.242:6379
"value4"
192.168.75.242:6379> get key5
-> Redirected to slot [9057] located at 192.168.75.241:6379
"value5"
192.168.75.241:6379> get key6
-> Redirected to slot [4866] located at 192.168.75.241:6380
"value6"
192.168.75.241:6380> get key7
-> Redirected to slot [803] located at 192.168.75.240:6381
"value7"
192.168.75.240:6381> get key8
-> Redirected to slot [13004] located at 192.168.75.242:6379
"value8"
192.168.75.242:6379> get key9
-> Redirected to slot [8941] located at 192.168.75.241:6379
"value9"
192.168.75.241:6379> get key10
-> Redirected to slot [5850] located at 192.168.75.240:6381
"value10"
192.168.75.240:6381> 

我们发现,key/value全部存在,但和集群新增节点前相比,key3、key7、key10的哈希槽所属节点发生了变化

删除节点

删除从节点192.168.75.240:6382,他的节点ID为:069f4ff213f5ad8f2bd59424eb25e1bf7e3eed12

redis-cli --cluster del-node 192.168.75.240:6379 069f4ff213f5ad8f2bd59424eb25e1bf7e3eed12

执行结果

[root@localhost ~]# redis-cli --cluster del-node 192.168.75.240:6379 069f4ff213f5ad8f2bd59424eb25e1bf7e3eed12
>>> Removing node 069f4ff213f5ad8f2bd59424eb25e1bf7e3eed12 from cluster 192.168.75.240:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

存在哈希槽的主节点不能直接删除,所以我们先移动主节点192.168.75.240:6381哈希槽至其他三个主节点
当前主节点哈希槽状态:0-1364 5461-6826 10923-12287
共有哈希槽=1365 + 1366 + 1365 = 4096个

redis-cli --cluster reshard 192.168.75.240:6379 --cluster-from b7352e36c147dcba86cdc1103a066f5e95606eac --cluster-to 726b7ba73ff0e4849a13fe1612ebc952d1c8786c --cluster-slots 1365 --cluster-yes
redis-cli --cluster reshard 192.168.75.240:6379 --cluster-from b7352e36c147dcba86cdc1103a066f5e95606eac --cluster-to 19f46bcef63286be0ebbcfd2acb9beae31f471d4 --cluster-slots 1366 --cluster-yes
redis-cli --cluster reshard 192.168.75.240:6379 --cluster-from b7352e36c147dcba86cdc1103a066f5e95606eac --cluster-to afad77c65770a962e53b89b275578a5ef382f039 --cluster-slots 1365 --cluster-yes

--cluster-from要移走哈希槽的节点ID
--cluster-to要移入哈希槽的节点ID
--cluster-slots要移动的哈希槽数
查看集群节点信息

redis-cli -c -p 6379 cluster nodes
c36e8e3175f53cccdf8640424192593c230ebd89 192.168.75.240:6380@16380 slave afad77c65770a962e53b89b275578a5ef382f039 0 1590301135000 12 connected
b16f676efe8d7998bfe56386434895ced77e79e7 192.168.75.240:6379@16379 myself,slave 726b7ba73ff0e4849a13fe1612ebc952d1c8786c 0 1590301134000 1 connected
afad77c65770a962e53b89b275578a5ef382f039 192.168.75.242:6379@16379 master - 0 1590301131000 12 connected 10923-16383
19f46bcef63286be0ebbcfd2acb9beae31f471d4 192.168.75.241:6379@16379 master - 0 1590301135000 11 connected 5461-10922
096e85ce032914c9c0f3107b6d3c49e612a57ab7 192.168.75.242:6380@16380 slave 19f46bcef63286be0ebbcfd2acb9beae31f471d4 0 1590301137033 11 connected
726b7ba73ff0e4849a13fe1612ebc952d1c8786c 192.168.75.241:6380@16380 master - 0 1590301134929 10 connected 0-5460
b7352e36c147dcba86cdc1103a066f5e95606eac 192.168.75.240:6381@16381 master - 0 1590301135960 9 connected

我们看到192.168.75.240:6381上已经没有了哈希槽
删除主节点

redis-cli --cluster del-node 192.168.75.240:6379 b7352e36c147dcba86cdc1103a066f5e95606eac

执行结果

[root@localhost ~]# redis-cli --cluster del-node 192.168.75.240:6379 b7352e36c147dcba86cdc1103a066f5e95606eac
>>> Removing node b7352e36c147dcba86cdc1103a066f5e95606eac from cluster 192.168.75.240:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

查看集群节点信息

redis-cli -c -p 6379 cluster nodes
c36e8e3175f53cccdf8640424192593c230ebd89 192.168.75.240:6380@16380 slave afad77c65770a962e53b89b275578a5ef382f039 0 1590301451939 12 connected
b16f676efe8d7998bfe56386434895ced77e79e7 192.168.75.240:6379@16379 myself,slave 726b7ba73ff0e4849a13fe1612ebc952d1c8786c 0 1590301451000 1 connected
afad77c65770a962e53b89b275578a5ef382f039 192.168.75.242:6379@16379 master - 0 1590301453008 12 connected 10923-16383
19f46bcef63286be0ebbcfd2acb9beae31f471d4 192.168.75.241:6379@16379 master - 0 1590301452000 11 connected 5461-10922
096e85ce032914c9c0f3107b6d3c49e612a57ab7 192.168.75.242:6380@16380 slave 19f46bcef63286be0ebbcfd2acb9beae31f471d4 0 1590301451000 11 connected
726b7ba73ff0e4849a13fe1612ebc952d1c8786c 192.168.75.241:6380@16380 master - 0 1590301454058 10 connected 0-5460

现在依然是三主三从了
再次查看key/value是否存在

[root@localhost ~]# redis-cli -c -p 6379
127.0.0.1:6379> get key1
-> Redirected to slot [9189] located at 192.168.75.241:6379
"value1"
192.168.75.241:6379> get key2
-> Redirected to slot [4998] located at 192.168.75.241:6380
"value2"
192.168.75.241:6380> get key3
"value3"
192.168.75.241:6380> get key4
-> Redirected to slot [13120] located at 192.168.75.242:6379
"value4"
192.168.75.242:6379> get key5
-> Redirected to slot [9057] located at 192.168.75.241:6379
"value5"
192.168.75.241:6379> get key6
-> Redirected to slot [4866] located at 192.168.75.241:6380
"value6"
192.168.75.241:6380> get key7
"value7"
192.168.75.241:6380> get key8
-> Redirected to slot [13004] located at 192.168.75.242:6379
"value8"
192.168.75.242:6379> get key9
-> Redirected to slot [8941] located at 192.168.75.241:6379
"value9"
192.168.75.241:6379> get key10
"value10"
192.168.75.241:6379> 

之前设置的key/value依然存在,值得一提的是

127.0.0.1:6379> mget key1 key2 key3 key4 key5 key6 key7 key8 key9 key10
(error) CROSSSLOT Keys in request don't hash to the same slot
127.0.0.1:6379> 

mget和mset命令的key必须得在同一个哈希槽上,跨哈希槽不允许使用,所以mget和mset命令不建议在Redis cluster集群中使用。