现象
gitlab部分操作报错500。通过Rails日志发现以下报错:
报错:
RedisClient::CommandError
ERR unknown command 'HELLO'
{
"severity": "ERROR",
"time": "2024-04-22T02:50:16.906Z",
"correlation_id": "01HW1VC67EJEGKQ1A1CW9TMVPV",
"meta.caller_id": "UserSettings::PersonalAccessTokensController#create",
"meta.remote_ip": "10.1.11.11",
"meta.feature_category": "system_access",
"meta.user": "root",
"meta.user_id": 1,
"meta.client_id": "user/1",
"exception.class": "RedisClient::CommandError",
"exception.message": "ERR unknown command 'HELLO'",
"exception.backtrace": [
"lib/gitlab/instrumentation/redis_client_middleware.rb:26:in `block in call_pipelined'",
"lib/gitlab/instrumentation/redis_helper.rb:17:in `instrument_call'",
"lib/gitlab/instrumentation/redis_client_middleware.rb:25:in `call_pipelined'",
"lib/gitlab/patch/redis_client.rb:12:in `ensure_connected'",
"config/initializers/forbid_sidekiq_in_transactions.rb:82:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'",
"app/services/notification_service.rb:95:in `access_token_created'",
"app/services/personal_access_tokens/create_service.rb:20:in `execute'",
"app/controllers/user_settings/personal_access_tokens_controller.rb:33:in `create'",
"app/controllers/application_controller.rb:468:in `set_current_admin'",
"lib/gitlab/session.rb:11:in `with_session'",
"app/controllers/application_controller.rb:459:in `set_session_storage'",
"lib/gitlab/i18n.rb:114:in `with_locale'",
"lib/gitlab/i18n.rb:120:in `with_user_locale'",
"app/controllers/application_controller.rb:450:in `set_locale'",
"app/controllers/application_controller.rb:443:in `set_current_context'",
"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
"lib/gitlab/metrics/web_transaction.rb:46:in `run'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/go.rb:20:in `call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:40:in `within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
"lib/gitlab/middleware/multipart.rb:173:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"lib/gitlab/middleware/path_traversal_check.rb:35:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:15:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:44:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'",
"lib/gitlab/middleware/release_env.rb:13:in `call'"
],
"user.username": "root",
"tags.program": "web",
"tags.locale": "en",
"tags.feature_category": "system_access",
"tags.correlation_id": "01HW1VC67EJEGKQ1A1CW9TMVPV",
"extra.storage": "queues"
}
排查
日志报错是Redis执行HELLO命令报错,这个命令适用于指定RESP版本。
- Gitlab官方文档要求16.0以上版本,需要使用Redis 6.x or 7.x
- Redis 6.x 开始支持RESP3
- 连接到Redis,执行Hello 2成功;执行Hello 3报错:ERR unknown command 'HELLO'
结合以上3点,怀疑Gitlab 16.x版本之前使用RESP2,之后使用RESP3。
进入Redis,执行monitor,复现问题查看操作:
再次看到与HELLO 3命令相关。
结论&后续处理
结论(踩坑)文章来源:https://www.toymoban.com/news/detail-858508.html
- 部分公有云Redis 6.x版本,不支持RESP3协议。
- Gitlab官方虽然明确说明16.x要使用Redis 6.x or 7.x,但是没有阐述原因。我想其中一个原因就是新的ruby代码对于RESP3协议的依赖。如果官网明确说明,并且附上前置检查步骤会更好。
后续处理文章来源地址https://www.toymoban.com/news/detail-858508.html
- 可以自己部署6.x版本redis高可用节点。
- 可以选择支持RESP3协议的公有云Redis实例使用。
到了这里,关于gitlab 16.x - ERR unknown command ‘HELLO‘的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!