Redis 分批删除字符串
- 有时候我们需要清理一些非常大的key（如hash键），或者通配非常多的（如string类型）， 如果直接使用keys、del操作会对线上的redis有性能影响，一般建议使用unlink 异步删除操作，释放交给redis自身去处理，但也有一些场景，可能需要快速释放内存，或者通配去删除 ，或者针对哈希表的某些item进行删除，这时候可以通过 SCAN 操作，再匹配后删除。
- 删除字符串时，同时要注意字符串ket的内存使用是否过大，删除过大也会影响到redis性能，下面提供按批量的删除方法，在删除时观察监控指标， CPU利用率是否在合理范围
Python
使用python 的 redis 模块，可以对sentinel 、 cluster 模式创建客户端
from redis.sentinel import Sentinel
from redis.cluster import RedisCluster, ClusterNode
通过SCAN的方式来删除， 设置 内存大小 10KB （按实际调整大小）， match 为匹配的键，比如： user* ， redis_server为创建的客户端，可以通过 Sentinel类、 RedisCluster 类 和实例化对象。 time.sleep控制每次操作间隔，建议可以先每500ms清理1000个，观察性能再做适当调整。代码参考如下：
def del_by_scan(redis_server, match, count, memory_usage=10240):
cursor = '0'
success, skip = 0, 0
print(f"begin to del_by_scan={match}")
while cursor != 0:
cursor, data = redis_server.scan(cursor=cursor, match=match, count=count)
for key in data:
memory = redis_server.memory_usage(key)
if memory > memory_usage:
print(f"scan key: {key}, type({key}): {redis_server.type(key)}, memory usage: {memory}B > 1.2MBB, skip.")
skip += 1
else:
print(f"scan key: {key}, type({key}): {redis_server.type(key)}, memory usage: {memory}B will be delete.")
redis_server.delete(key)
success += 1
time.sleep(1)
print(f"delete scan key: {match}, Done ! success: {success}, skip: {skip} ")
插入1000条测试数据，然后执行批量删除 user* 开头的 string 类型，每1秒删除100个（仅测试），实际可以调大些（在保证性能前提下） ， 测试效果如下：memory_usage=49用于测试调很小。实际建议10KB左右。
del_by_scan(redis_server, "user*", 100， memory_usage=49)
删除后执行日志如下：成功删除 999条。 跳过1条（超过设置的大小）。
2023-07-07 13:27:13:begin to del_by_scan=user*
2023-07-07 13:27:13:scan key: b'user991', type(b'user991'): b'string', memory usage: 49B will be delete.
2023-07-07 13:27:13:scan key: b'user589', type(b'user589'): b'string', memory usage: 49B will be delete.
2023-07-07 13:27:13:scan key: b'user685', type(b'user685'): b'string', memory usage: 49B will be delete.
......
2023-07-07 13:27:19:scan key: b'user1000', type(b'user1000'): b'string', memory usage: 50B > 49B, skip.
......
2023-07-07 13:27:25:delete scan key: user*, Done ! success: 999, skip: 1
后端调用是Redis SCAN 命令，参考如下：
1688706474.976630 [0 192.168.88.1:65060] "SCAN" "0" "MATCH" "user*" "COUNT" "100"
1688706475.020499 [0 192.168.88.1:65060] "SCAN" "152" "MATCH" "user*" "COUNT" "100"
1688706475.077715 [0 192.168.88.1:65060] "SCAN" "972" "MATCH" "user*" "COUNT" "100"
1688706475.124888 [0 192.168.88.1:65060] "SCAN" "850" "MATCH" "user*" "COUNT" "100"
1688706475.178739 [0 192.168.88.1:65060] "SCAN" "230" "MATCH" "user*" "COUNT" "100"
1688706475.229983 [0 192.168.88.1:65060] "SCAN" "702" "MATCH" "user*" "COUNT" "100"
1688706475.281272 [0 192.168.88.1:65060] "SCAN" "873" "MATCH" "user*" "COUNT" "100"
1688706475.338270 [0 192.168.88.1:65060] "SCAN" "653" "MATCH" "user*" "COUNT" "100"
1688706475.390810 [0 192.168.88.1:65060] "SCAN" "83" "MATCH" "user*" "COUNT" "100"
1688706475.443487 [0 192.168.88.1:65060] "SCAN" "71" "MATCH" "user*" "COUNT" "100"
1688706475.501230 [0 192.168.88.1:65060] "SCAN" "191" "MATCH" "user*" "COUNT" "100"
Go
使用Go Redis支持Redis Server和Redis Cluster的Golang客户端的: “github.com/redis/go-redis/v9” 处理方式和Python相似，都是通过SCAN扫描后执行分批删除。 代码参考如下：
func scanKey(ctx context.Context, rdb *redis.Client, pattern string, batchSize int64, memoryUsage int64) error {
cursor, total, success, skip := uint64(0), uint(0), uint(0), uint(0)
var err error
addr := rdb.Options().Addr
for {
var result []string
result, cursor, err = rdb.Scan(ctx, cursor, pattern, batchSize).Result()
if err != nil {
log.Printf("could not scan: %q\n", err)
return err
}
total += uint(len(result))
for i := 0; i < len(result); i++ {
key := result[i]
keyType := rdb.Type(ctx, key).Val()
memory := rdb.MemoryUsage(ctx, key).Val()
if memory > memoryUsage {
log.Printf("redis: %s, scan key: %s, type: %s, memory usage: %dB > %dB, skip.", addr, key, keyType, memory, memoryUsage)
skip++
} else {
if _, err = rdb.Del(ctx, key).Result(); err != nil {
log.Printf("redis: %s, delete string key: %s failed: %q", addr, key, err)
return err
}
log.Printf("redis: %s, key: %s, type: %s, memory usage: %dB, deleted.", addr, key, keyType, memory)
success++
}
}
if cursor == 0 {
break
}
time.Sleep(1000 * time.Millisecond)
}
log.Printf("redis: %s, All scan key=%s Done ! total: {%d}, success: {%d}, skip: {%d}", addr, pattern, total, success, skip)
return err
}
调用删除函数，传入相关参数
client := redis.NewClient(&redis.Options{
Addr: address,
Password: password,
DB: db,
ReadTimeout: 1 * time.Minute,
})
scanKey(ctx, client, "user*", 300, 49)
执行删除，相关日志参考如下：文章来源：https://www.toymoban.com/news/detail-556981.html
2023/07/07 13:07:56 redis server connect success: "PONG", address: "192.168.88.11:6379"
2023/07/07 13:07:56 redis: 192.168.88.11:6379, key: user569, type: string, memory usage: 49B, deleted.
2023/07/07 13:07:56 redis: 192.168.88.11:6379, key: user627, type: string, memory usage: 49B, deleted.
2023/07/07 13:07:56 redis: 192.168.88.11:6379, key: user436, type: string, memory usage: 49B, deleted.
......
2023/07/07 14:29:51 redis: 192.168.88.11:6379, scan key: user1000, type: string, memory usage: 50B > 49B, skip.
......
2023/07/07 14:29:54 redis: 192.168.88.11:6379, All scan key=user* Done ! total: {1000}, success: {999}, skip: {1}
后端同样调用SCAN命令，参考如下文章来源地址https://www.toymoban.com/news/detail-556981.html
1688712114.495473 [0 192.168.88.11:46774] "scan" "0" "match" "user*" "count" "300"
1688712115.620560 [0 192.168.88.11:46774] "scan" "594" "match" "user*" "count" "300"
1688712116.758503 [0 192.168.88.11:46774] "scan" "617" "match" "user*" "count" "300"
1688712117.888501 [0 192.168.88.11:46774] "scan" "391" "match" "user*" "count" "300"
总结
- 对于大批量的key做删除，或者针对某几个大key删除（如HASH， SET ）类型等的操作，建议使用分批删除
- 删除时控制每批的数量、控制间隔时间，字符串同时考虑其内存大小，删除过大时会影响到redis性能
- 对于非字符串，其它类型如HASH也提供了HSCAN方式扫描元素，操作方式同上
到了这里，关于Redis 分批删除字符串键操作的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！
