本篇笔记记录了Redis Cluster集群(扩容)新增节点、(缩容)删除节点的方法，以及如何在(扩容)新增和(缩容)删除节点时对hash slot(哈希槽)进行重新分配
Redis集群搭建请参考 Redis Cluster集群搭建
当前Redis集群节点
192.168.75.240:6379 192.168.75.240:6380
192.168.75.241:6379 192.168.75.241:6380
192.168.75.242:6379 192.168.75.242:6380
什么是hash slot(哈希槽)
援引作者说法：
Redis Cluster没有使用一致哈希，而是使用一种不同形式的分片，每个key从概念上讲都是我们称为hash slot的一部分。
Redis Cluster中有16384个hash slot，要计算某个key所属的hash slot，我们只需对key的CRC16值和16384取模。
简洁的说，一致性哈希计算key分布使用Time33(也叫DJBX33A)算法，Redis Cluster计算key分布使用CRC16结果和16384取模；更简洁的说，一致性哈希将key分布在2的32次方个节点上，Redis Cluster将key分布在16384个哈希槽上，Redis Cluster集群中每个主节点分布一个哈希槽的话，Redis Cluster集群最多容纳16384个主节点，当然，实际应用中不可能有那么多主节点。
查看当前集群节点和哈希槽分布
登录任一节点
redis-cli -c -p 6379
-c表示集群
执行命令
cluster nodes
集群节点信息如下
127.0.0.1:6379> cluster nodes
c36e8e3175f53cccdf8640424192593c230ebd89 192.168.75.240:6380@16380 slave afad77c65770a962e53b89b275578a5ef382f039 0 1590295653703 5 connected
b16f676efe8d7998bfe56386434895ced77e79e7 192.168.75.240:6379@16379 myself,slave 726b7ba73ff0e4849a13fe1612ebc952d1c8786c 0 1590295652000 1 connected
afad77c65770a962e53b89b275578a5ef382f039 192.168.75.242:6379@16379 master - 0 1590295656000 5 connected 10923-16383
19f46bcef63286be0ebbcfd2acb9beae31f471d4 192.168.75.241:6379@16379 master - 0 1590295654745 3 connected 5461-10922
096e85ce032914c9c0f3107b6d3c49e612a57ab7 192.168.75.242:6380@16380 slave 19f46bcef63286be0ebbcfd2acb9beae31f471d4 0 1590295655000 6 connected
726b7ba73ff0e4849a13fe1612ebc952d1c8786c 192.168.75.241:6380@16380 master - 0 1590295656871 7 connected 0-5460
- 第一列为节点ID
- 第二列为节点IP:端口号@集群管理端口号
- 第三列为主/从节点，myself表示当前执行命令的节点
- 第四列为父节点ID，主节点的父节点ID为0
- 主节点的最后一列为哈希槽分布区间，从节点同步主节点数据，槽位依据主节点槽位
设置key/value
127.0.0.1:6379> set key1 value1
-> Redirected to slot [9189] located at 192.168.75.241:6379
OK
192.168.75.241:6379> set key2 value2
-> Redirected to slot [4998] located at 192.168.75.241:6380
OK
192.168.75.241:6380> set key3 value3
OK
192.168.75.241:6380> set key4 value4
-> Redirected to slot [13120] located at 192.168.75.242:6379
OK
192.168.75.242:6379> set key5 value5
-> Redirected to slot [9057] located at 192.168.75.241:6379
OK
192.168.75.241:6379> set key6 value6
-> Redirected to slot [4866] located at 192.168.75.241:6380
OK
192.168.75.241:6380> set key7 value7
OK
192.168.75.241:6380> set key8 value8
-> Redirected to slot [13004] located at 192.168.75.242:6379
OK
192.168.75.242:6379> set key9 value9
-> Redirected to slot [8941] located at 192.168.75.241:6379
OK
192.168.75.241:6379> set key10 value10
OK
我设置了10个key/value，结果显示了key对应的哈希槽和redis节点IP:端口号信息
新增节点并分配哈希槽
任意服务器创建主节点和从节点，可以不在一台服务器上，为省事我选择在192.168.75.240上操作
创建Redis工作目录
mkdir -p /usr/local/redis/6381
mkdir -p /usr/local/redis/6382
复制安装包下配置文件到安装目录
cp /usr/local/src/redis-5.0.5/redis.conf /usr/local/redis/redis_6381.conf
cp /usr/local/src/redis-5.0.5/redis.conf /usr/local/redis/redis_6382.conf
修改配置文件
redis_6381.conf
bind 0.0.0.0
protected-mode no
port 6381
daemonize yes
pidfile /var/run/redis_6381.pid
logfile "/var/log/redis/6381.log"
dir /usr/local/redis/6381
cluster-enabled yes
cluster-config-file nodes-6381.conf
cluster-node-timeout 15000
redis_6382.conf
bind 0.0.0.0
protected-mode no
port 6382
daemonize yes
pidfile /var/run/redis_6382.pid
logfile "/var/log/redis/6382.log"
dir /usr/local/redis/6382
cluster-enabled yes
cluster-config-file nodes-6382.conf
cluster-node-timeout 15000
启动Redis
redis-server /usr/local/redis/redis_6381.conf
redis-server /usr/local/redis/redis_6382.conf
将两个节点加入集群
添加192.168.75.240:6381到192.168.75.240:6379所属的集群中作为master
redis-cli --cluster add-node 192.168.75.240:6381 192.168.75.240:6379
查看集群节点信息
redis-cli -c -p 6379 cluster nodes
c36e8e3175f53cccdf8640424192593c230ebd89 192.168.75.240:6380@16380 slave afad77c65770a962e53b89b275578a5ef382f039 0 1590297292519 5 connected
b16f676efe8d7998bfe56386434895ced77e79e7 192.168.75.240:6379@16379 myself,slave 726b7ba73ff0e4849a13fe1612ebc952d1c8786c 0 1590297291000 1 connected
afad77c65770a962e53b89b275578a5ef382f039 192.168.75.242:6379@16379 master - 0 1590297293564 5 connected 10923-16383
19f46bcef63286be0ebbcfd2acb9beae31f471d4 192.168.75.241:6379@16379 master - 0 1590297293000 3 connected 5461-10922
096e85ce032914c9c0f3107b6d3c49e612a57ab7 192.168.75.242:6380@16380 slave 19f46bcef63286be0ebbcfd2acb9beae31f471d4 0 1590297294599 6 connected
726b7ba73ff0e4849a13fe1612ebc952d1c8786c 192.168.75.241:6380@16380 master - 0 1590297292000 7 connected 0-5460
b7352e36c147dcba86cdc1103a066f5e95606eac 192.168.75.240:6381@16381 master - 0 1590297291498 0 connected
192.168.75.240:6381的ID为b7352e36c147dcba86cdc1103a066f5e95606eac
添加192.168.75.240:6382为192.168.75.240:6381的从节点到192.168.75.240:6379所属的集群
redis-cli --cluster add-node 192.168.75.240:6382 192.168.75.240:6379 --cluster-slave --cluster-master-id b7352e36c147dcba86cdc1103a066f5e95606eac
查看集群节点信息
redis-cli -c -p 6379 cluster nodes
069f4ff213f5ad8f2bd59424eb25e1bf7e3eed12 192.168.75.240:6382@16382 slave b7352e36c147dcba86cdc1103a066f5e95606eac 0 1590297530000 0 connected
c36e8e3175f53cccdf8640424192593c230ebd89 192.168.75.240:6380@16380 slave afad77c65770a962e53b89b275578a5ef382f039 0 1590297528000 5 connected
b16f676efe8d7998bfe56386434895ced77e79e7 192.168.75.240:6379@16379 myself,slave 726b7ba73ff0e4849a13fe1612ebc952d1c8786c 0 1590297524000 1 connected
afad77c65770a962e53b89b275578a5ef382f039 192.168.75.242:6379@16379 master - 0 1590297531469 5 connected 10923-16383
19f46bcef63286be0ebbcfd2acb9beae31f471d4 192.168.75.241:6379@16379 master - 0 1590297528000 3 connected 5461-10922
096e85ce032914c9c0f3107b6d3c49e612a57ab7 192.168.75.242:6380@16380 slave 19f46bcef63286be0ebbcfd2acb9beae31f471d4 0 1590297529349 6 connected
726b7ba73ff0e4849a13fe1612ebc952d1c8786c 192.168.75.241:6380@16380 master - 0 1590297530402 7 connected 0-5460
b7352e36c147dcba86cdc1103a066f5e95606eac 192.168.75.240:6381@16381 master - 0 1590297528319 0 connected
我们看到，192.168.75.240:6382以从节点身份加入到了集群，但新增的主节点192.168.75.240:6381还没有分配哈希槽
重新分配哈希槽
自动平衡集群中的各节点哈希槽数
redis-cli --cluster rebalance --cluster-use-empty-masters 192.168.75.240:6379
执行过程
>>> Performing Cluster Check (using node 192.168.75.240:6379)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Rebalancing across 4 nodes. Total weight = 4.00
Moving 1366 slots from 192.168.75.241:6379 to 192.168.75.240:6381
###########################......
Moving 1365 slots from 192.168.75.241:6380 to 192.168.75.240:6381
###########################......
Moving 1365 slots from 192.168.75.242:6379 to 192.168.75.240:6381
###########################......
我们看到，系统自动从当前各主节点中拿出一定数量的哈希槽，转义至新的主节点中，以达到哈希槽平衡，Moving 1365 slots中的1365为移动的哈希槽数量
查看集群节点信息
redis-cli -c -p 6379 cluster nodes
069f4ff213f5ad8f2bd59424eb25e1bf7e3eed12 192.168.75.240:6382@16382 slave b7352e36c147dcba86cdc1103a066f5e95606eac 0 1590298680718 9 connected
c36e8e3175f53cccdf8640424192593c230ebd89 192.168.75.240:6380@16380 slave afad77c65770a962e53b89b275578a5ef382f039 0 1590298678000 5 connected
b16f676efe8d7998bfe56386434895ced77e79e7 192.168.75.240:6379@16379 myself,slave 726b7ba73ff0e4849a13fe1612ebc952d1c8786c 0 1590298681000 1 connected
afad77c65770a962e53b89b275578a5ef382f039 192.168.75.242:6379@16379 master - 0 1590298676528 5 connected 12288-16383
19f46bcef63286be0ebbcfd2acb9beae31f471d4 192.168.75.241:6379@16379 master - 0 1590298681000 3 connected 6827-10922
096e85ce032914c9c0f3107b6d3c49e612a57ab7 192.168.75.242:6380@16380 slave 19f46bcef63286be0ebbcfd2acb9beae31f471d4 0 1590298679000 6 connected
726b7ba73ff0e4849a13fe1612ebc952d1c8786c 192.168.75.241:6380@16380 master - 0 1590298680000 7 connected 1365-5460
b7352e36c147dcba86cdc1103a066f5e95606eac 192.168.75.240:6381@16381 master - 0 1590298681766 9 connected 0-1364 5461-6826 10923-12287
我们看到，主节点192.168.75.240:6381已经分配了哈希槽
那么我们刚才设置的key/value还在吗，我们测试一下
[root@localhost ~]# redis-cli -c -p 6379
127.0.0.1:6379> get key1
-> Redirected to slot [9189] located at 192.168.75.241:6379
"value1"
192.168.75.241:6379> get key2
-> Redirected to slot [4998] located at 192.168.75.241:6380
"value2"
192.168.75.241:6380> get key3
-> Redirected to slot [935] located at 192.168.75.240:6381
"value3"
192.168.75.240:6381> get key4
-> Redirected to slot [13120] located at 192.168.75.242:6379
"value4"
192.168.75.242:6379> get key5
-> Redirected to slot [9057] located at 192.168.75.241:6379
"value5"
192.168.75.241:6379> get key6
-> Redirected to slot [4866] located at 192.168.75.241:6380
"value6"
192.168.75.241:6380> get key7
-> Redirected to slot [803] located at 192.168.75.240:6381
"value7"
192.168.75.240:6381> get key8
-> Redirected to slot [13004] located at 192.168.75.242:6379
"value8"
192.168.75.242:6379> get key9
-> Redirected to slot [8941] located at 192.168.75.241:6379
"value9"
192.168.75.241:6379> get key10
-> Redirected to slot [5850] located at 192.168.75.240:6381
"value10"
192.168.75.240:6381>
我们发现，key/value全部存在，但和集群新增节点前相比，key3、key7、key10的哈希槽所属节点发生了变化
删除节点
删除从节点192.168.75.240:6382，他的节点ID为：069f4ff213f5ad8f2bd59424eb25e1bf7e3eed12
redis-cli --cluster del-node 192.168.75.240:6379 069f4ff213f5ad8f2bd59424eb25e1bf7e3eed12
执行结果
[root@localhost ~]# redis-cli --cluster del-node 192.168.75.240:6379 069f4ff213f5ad8f2bd59424eb25e1bf7e3eed12
>>> Removing node 069f4ff213f5ad8f2bd59424eb25e1bf7e3eed12 from cluster 192.168.75.240:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
存在哈希槽的主节点不能直接删除，所以我们先移动主节点192.168.75.240:6381哈希槽至其他三个主节点
当前主节点哈希槽状态：0-1364 5461-6826 10923-12287
共有哈希槽=1365 + 1366 + 1365 = 4096个
redis-cli --cluster reshard 192.168.75.240:6379 --cluster-from b7352e36c147dcba86cdc1103a066f5e95606eac --cluster-to 726b7ba73ff0e4849a13fe1612ebc952d1c8786c --cluster-slots 1365 --cluster-yes
redis-cli --cluster reshard 192.168.75.240:6379 --cluster-from b7352e36c147dcba86cdc1103a066f5e95606eac --cluster-to 19f46bcef63286be0ebbcfd2acb9beae31f471d4 --cluster-slots 1366 --cluster-yes
redis-cli --cluster reshard 192.168.75.240:6379 --cluster-from b7352e36c147dcba86cdc1103a066f5e95606eac --cluster-to afad77c65770a962e53b89b275578a5ef382f039 --cluster-slots 1365 --cluster-yes
--cluster-from要移走哈希槽的节点ID
--cluster-to要移入哈希槽的节点ID
--cluster-slots要移动的哈希槽数
查看集群节点信息
redis-cli -c -p 6379 cluster nodes
c36e8e3175f53cccdf8640424192593c230ebd89 192.168.75.240:6380@16380 slave afad77c65770a962e53b89b275578a5ef382f039 0 1590301135000 12 connected
b16f676efe8d7998bfe56386434895ced77e79e7 192.168.75.240:6379@16379 myself,slave 726b7ba73ff0e4849a13fe1612ebc952d1c8786c 0 1590301134000 1 connected
afad77c65770a962e53b89b275578a5ef382f039 192.168.75.242:6379@16379 master - 0 1590301131000 12 connected 10923-16383
19f46bcef63286be0ebbcfd2acb9beae31f471d4 192.168.75.241:6379@16379 master - 0 1590301135000 11 connected 5461-10922
096e85ce032914c9c0f3107b6d3c49e612a57ab7 192.168.75.242:6380@16380 slave 19f46bcef63286be0ebbcfd2acb9beae31f471d4 0 1590301137033 11 connected
726b7ba73ff0e4849a13fe1612ebc952d1c8786c 192.168.75.241:6380@16380 master - 0 1590301134929 10 connected 0-5460
b7352e36c147dcba86cdc1103a066f5e95606eac 192.168.75.240:6381@16381 master - 0 1590301135960 9 connected
我们看到192.168.75.240:6381上已经没有了哈希槽
删除主节点
redis-cli --cluster del-node 192.168.75.240:6379 b7352e36c147dcba86cdc1103a066f5e95606eac
执行结果
[root@localhost ~]# redis-cli --cluster del-node 192.168.75.240:6379 b7352e36c147dcba86cdc1103a066f5e95606eac
>>> Removing node b7352e36c147dcba86cdc1103a066f5e95606eac from cluster 192.168.75.240:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
查看集群节点信息
redis-cli -c -p 6379 cluster nodes
c36e8e3175f53cccdf8640424192593c230ebd89 192.168.75.240:6380@16380 slave afad77c65770a962e53b89b275578a5ef382f039 0 1590301451939 12 connected
b16f676efe8d7998bfe56386434895ced77e79e7 192.168.75.240:6379@16379 myself,slave 726b7ba73ff0e4849a13fe1612ebc952d1c8786c 0 1590301451000 1 connected
afad77c65770a962e53b89b275578a5ef382f039 192.168.75.242:6379@16379 master - 0 1590301453008 12 connected 10923-16383
19f46bcef63286be0ebbcfd2acb9beae31f471d4 192.168.75.241:6379@16379 master - 0 1590301452000 11 connected 5461-10922
096e85ce032914c9c0f3107b6d3c49e612a57ab7 192.168.75.242:6380@16380 slave 19f46bcef63286be0ebbcfd2acb9beae31f471d4 0 1590301451000 11 connected
726b7ba73ff0e4849a13fe1612ebc952d1c8786c 192.168.75.241:6380@16380 master - 0 1590301454058 10 connected 0-5460
现在依然是三主三从了
再次查看key/value是否存在
[root@localhost ~]# redis-cli -c -p 6379
127.0.0.1:6379> get key1
-> Redirected to slot [9189] located at 192.168.75.241:6379
"value1"
192.168.75.241:6379> get key2
-> Redirected to slot [4998] located at 192.168.75.241:6380
"value2"
192.168.75.241:6380> get key3
"value3"
192.168.75.241:6380> get key4
-> Redirected to slot [13120] located at 192.168.75.242:6379
"value4"
192.168.75.242:6379> get key5
-> Redirected to slot [9057] located at 192.168.75.241:6379
"value5"
192.168.75.241:6379> get key6
-> Redirected to slot [4866] located at 192.168.75.241:6380
"value6"
192.168.75.241:6380> get key7
"value7"
192.168.75.241:6380> get key8
-> Redirected to slot [13004] located at 192.168.75.242:6379
"value8"
192.168.75.242:6379> get key9
-> Redirected to slot [8941] located at 192.168.75.241:6379
"value9"
192.168.75.241:6379> get key10
"value10"
192.168.75.241:6379>
之前设置的key/value依然存在，值得一提的是
127.0.0.1:6379> mget key1 key2 key3 key4 key5 key6 key7 key8 key9 key10
(error) CROSSSLOT Keys in request don't hash to the same slot
127.0.0.1:6379>
mget和mset命令的key必须得在同一个哈希槽上，跨哈希槽不允许使用，所以mget和mset命令不建议在Redis cluster集群中使用。
再现理想