记录k8s kube-controller-manager-k8s-master kube-scheduler-k8s-master重启

这篇具有很好参考价值的文章主要介绍了记录k8s kube-controller-manager-k8s-master kube-scheduler-k8s-master重启。希望对大家有所帮助。如果存在错误或未考虑完全的地方,请大家不吝赐教,您也可以点击"举报违法"按钮提交疑问。

1、报错如下

I0529 01:47:12.679312       1 event.go:307] "Event occurred" object="k8s-node-1" fieldPath="" kind="Node" apiVersion="v1" type="Normal" reason="CIDRNotAvailable" message="Node k8s-node-1 status is now: CIDRNotAvailable"
E0529 01:48:44.516760       1 controller_utils.go:262] Error while processing Node Add/Delete: failed to allocate cidr from cluster cidr at idx:0: CIDR allocation failed; there are no remaining CIDRs left to allocate in the accepted range
I0529 01:48:44.516885       1 event.go:307] "Event occurred" object="k8s-master" fieldPath="" kind="Node" apiVersion="v1" type="Normal" reason="CIDRNotAvailable" message="Node k8s-master status is now: CIDRNotAvailable"
E0529 01:49:28.020461       1 controller_utils.go:262] Error while processing Node Add/Delete: failed to allocate cidr from cluster cidr at idx:0: CIDR allocation failed; there are no remaining CIDRs left to allocate in the accepted range
I0529 01:49:28.020839       1 event.go:307] "Event occurred" object="k8s-node-2" fieldPath="" kind="Node" apiVersion="v1" type="Normal" reason="CIDRNotAvailable" message="Node k8s-node-2 status is now: CIDRNotAvailable"

2、可能原因k8s初始化cluster-cidr和service-cluster-ip-range子网划分冲突

(原    --apiserver-advertise-address 10.19.3.15  --service-cidr 10.245.0.0/12    --pod-network-cidr 10.244.0.0/16)

3、修改/etc/kubernetes/manifests/kube-controller-manager.yaml

- --cluster-cidr=10.96.0.0/16                #修改此处

修改后不用重启组件或者集群,稍等后pod自动重建

 kubectl logs  kube-controller-manager-k8s-master  -n kube-system查看没有报错

(未修改之前,即使pod运行正常,查看pod日志还是有上述报错)

4、修改后没有发现问题,请大家在测试环境测试。

///

更新------------------新建测试环境依然存在类似问题

centos7  kube-1.27.1  containerd 1.6.19 (master为esxi虚拟机)

集群状态都正常,系统硬盘IO也正常。依照网上的加大心跳时间、修改硬盘参数,都不能解决。

最后问题可能出现在Esxi这个集群的存储上,虚拟机迁移硬盘至另外的存储主机。因为目前已经24小时ks和kc组件未重启了,还不能最后排除,但是大概率就是这儿的问题。

日志如下,希望可以帮助需要的人:

1、containerd报错

Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.394405539+08:00" level=info msg="StopPodSandbox for \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\""
Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.394537456+08:00" level=info msg="TearDown network for sandbox \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\" successfully"
Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.394591107+08:00" level=info msg="StopPodSandbox for \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\" returns successfully"
Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.395525689+08:00" level=info msg="RemovePodSandbox for \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\""
Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.395780557+08:00" level=info msg="Forcibly stopping sandbox \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\""
Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.396142546+08:00" level=info msg="TearDown network for sandbox \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\" successfully"
Jun 15 11:56:54 k8s-master containerd[34171]: time="2023-06-15T11:56:54.401661796+08:00" level=info msg="RemovePodSandbox \"219a26b5fa3428801e99f2fc9b801a503d547536f66b6e659b3e6083df9e9340\" returns successfully"
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.140016886+08:00" level=info msg="shim disconnected" id=8fd74e39f764d17344f0d5a0cfb92d6ade56421b0ce54d73d2971477d7a49eec
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.140641397+08:00" level=warning msg="cleaning up after shim disconnected" id=8fd74e39f764d17344f0d5a0cfb92d6ade56421b0ce54d73d2971477d7a49eec namespace=k8s.io
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.140835000+08:00" level=info msg="cleaning up dead shim"
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.152377826+08:00" level=warning msg="cleanup warnings time=\"2023-06-15T11:59:21+08:00\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=989621 runtime=io.containerd.runc.v2\n"
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.642769894+08:00" level=info msg="CreateContainer within sandbox \"2d27aa3e82f08d67ab9c6b8b821a324b86ba717b5d18c5729b382c488bd2f23f\" for container &ContainerMetadata{Name:kube-controller-manager,Attempt:1,}"
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.741636615+08:00" level=info msg="CreateContainer within sandbox \"2d27aa3e82f08d67ab9c6b8b821a324b86ba717b5d18c5729b382c488bd2f23f\" for &ContainerMetadata{Name:kube-controller-manager,Attempt:1,} returns container id \"53940ebcbf87d9313bb88b1656bc189745b04414888305a7ec47aef9c55fcdaf\""
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.742513549+08:00" level=info msg="StartContainer for \"53940ebcbf87d9313bb88b1656bc189745b04414888305a7ec47aef9c55fcdaf\""
Jun 15 11:59:21 k8s-master containerd[34171]: time="2023-06-15T11:59:21.855170471+08:00" level=info msg="StartContainer for \"53940ebcbf87d9313bb88b1656bc189745b04414888305a7ec47aef9c55fcdaf\" returns successfully"
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.392743400+08:00" level=info msg="shim disconnected" id=8eaef5a52f673266d0e141ae17a2d12ee377b7f08ad4a3f65d77f3abe0902c45
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.392850972+08:00" level=warning msg="cleaning up after shim disconnected" id=8eaef5a52f673266d0e141ae17a2d12ee377b7f08ad4a3f65d77f3abe0902c45 namespace=k8s.io
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.392869777+08:00" level=info msg="cleaning up dead shim"
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.405071189+08:00" level=warning msg="cleanup warnings time=\"2023-06-15T11:59:26+08:00\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=989732 runtime=io.containerd.runc.v2\n"
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.665183619+08:00" level=info msg="CreateContainer within sandbox \"24ebd020c4398151bdd87a97849fe02c4880f88bf132407e07ed5fad7c088932\" for container &ContainerMetadata{Name:kube-scheduler,Attempt:1,}"
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.719306795+08:00" level=info msg="CreateContainer within sandbox \"24ebd020c4398151bdd87a97849fe02c4880f88bf132407e07ed5fad7c088932\" for &ContainerMetadata{Name:kube-scheduler,Attempt:1,} returns container id \"e9f21ce36cace252a65544577beda1c6573e7473afc25d059df5d9234f18308b\""
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.720429540+08:00" level=info msg="StartContainer for \"e9f21ce36cace252a65544577beda1c6573e7473afc25d059df5d9234f18308b\""
Jun 15 11:59:26 k8s-master containerd[34171]: time="2023-06-15T11:59:26.834912477+08:00" level=info msg="StartContainer for \"e9f21ce36cace252a65544577beda1c6573e7473afc25d059df5d9234f18308b\" returns successfully"

2、etcd报错

{"level":"warn","ts":"2023-06-15T03:11:46.440Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"647.783649ms","expected-duration":"100ms","prefix":"","request":"header:<ID:1773161178989142284 username:\"kube-apiserver-etcd-client\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-controller-manager\" mod_revision:274019 > success:<request_put:<key:\"/registry/leases/kube-system/kube-controller-manager\" value_size:433 >> failure:<request_range:<key:\"/registry/leases/kube-system/kube-controller-manager\" > >>","response":"size:18"}
{"level":"info","ts":"2023-06-15T03:11:46.440Z","caller":"traceutil/trace.go:171","msg":"trace[1077021032] linearizableReadLoop","detail":"{readStateIndex:311409; appliedIndex:311408; }","duration":"393.007455ms","start":"2023-06-15T03:11:46.047Z","end":"2023-06-15T03:11:46.440Z","steps":["trace[1077021032] 'read index received'  (duration: 42.34µs)","trace[1077021032] 'applied index is now lower than readState.Index'  (duration: 392.963682ms)"],"step_count":2}
{"level":"warn","ts":"2023-06-15T03:11:46.440Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"393.171763ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/health\" ","response":"range_response_count:0 size:6"}
{"level":"info","ts":"2023-06-15T03:11:46.440Z","caller":"traceutil/trace.go:171","msg":"trace[2049818894] range","detail":"{range_begin:/registry/health; range_end:; response_count:0; response_revision:274023; }","duration":"393.252264ms","start":"2023-06-15T03:11:46.047Z","end":"2023-06-15T03:11:46.440Z","steps":["trace[2049818894] 'agreement among raft nodes before linearized reading'  (duration: 393.104647ms)"],"step_count":1}
{"level":"warn","ts":"2023-06-15T03:11:46.440Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-06-15T03:11:46.047Z","time spent":"393.363051ms","remote":"127.0.0.1:48410","response type":"/etcdserverpb.KV/Range","request count":0,"request size":18,"response count":0,"response size":30,"request content":"key:\"/registry/health\" "}
{"level":"info","ts":"2023-06-15T03:11:46.440Z","caller":"traceutil/trace.go:171","msg":"trace[499563569] transaction","detail":"{read_only:false; response_revision:274023; number_of_response:1; }","duration":"709.954696ms","start":"2023-06-15T03:11:45.730Z","end":"2023-06-15T03:11:46.440Z","steps":["trace[499563569] 'process raft request'  (duration: 61.343055ms)","trace[499563569] 'compare'  (duration: 647.534581ms)"],"step_count":2}
{"level":"warn","ts":"2023-06-15T03:11:46.440Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-06-15T03:11:45.730Z","time spent":"710.061185ms","remote":"127.0.0.1:48342","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":493,"response count":0,"response size":42,"request content":"compare:<target:MOD key:\"/registry/leases/kube-system/kube-controller-manager\" mod_revision:274019 > success:<request_put:<key:\"/registry/leases/kube-system/kube-controller-manager\" value_size:433 >> failure:<request_range:<key:\"/registry/leases/kube-system/kube-controller-manager\" > >"}
{"level":"info","ts":"2023-06-15T03:16:07.151Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":273955}
{"level":"info","ts":"2023-06-15T03:16:07.153Z","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":273955,"took":"1.552321ms","hash":1950790470}
{"level":"info","ts":"2023-06-15T03:16:07.153Z","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":1950790470,"revision":273955,"compact-revision":273469}
{"level":"info","ts":"2023-06-15T03:20:43.437Z","caller":"traceutil/trace.go:171","msg":"trace[126699866] transaction","detail":"{read_only:false; response_revision:274895; number_of_response:1; }","duration":"307.20387ms","start":"2023-06-15T03:20:43.130Z","end":"2023-06-15T03:20:43.437Z","steps":["trace[126699866] 'process raft request'  (duration: 307.036766ms)"],"step_count":1}
{"level":"warn","ts":"2023-06-15T03:20:43.438Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-06-15T03:20:43.130Z","time spent":"307.376032ms","remote":"127.0.0.1:48342","response type":"/etcdserverpb.KV/Txn","request count":1,"request size":465,"response count":0,"response size":42,"request content":"compare:<target:MOD key:\"/registry/leases/kube-system/kube-scheduler\" mod_revision:274893 > success:<request_put:<key:\"/registry/leases/kube-system/kube-scheduler\" value_size:414 >> failure:<request_range:<key:\"/registry/leases/kube-system/kube-scheduler\" > >"}
{"level":"warn","ts":"2023-06-15T03:21:05.415Z","caller":"etcdserver/v3_server.go:840","msg":"waiting for ReadIndex response took too long, retrying","sent-request-id":1773161178989145684,"retry-timeout":"500ms"}文章来源地址https://www.toymoban.com/news/detail-705774.html

到了这里,关于记录k8s kube-controller-manager-k8s-master kube-scheduler-k8s-master重启的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处: 如若内容造成侵权/违法违规/事实不符,请点击违法举报进行投诉反馈,一经查实,立即删除!

领支付宝红包 赞助服务器费用

相关文章

  • k8s安装kube-promethues(0.7版本)

    目录 k8s安装kube-promethues(0.7版本) 一.检查本地k8s版本,下载对应安装包 二.安装前准备 1.文件分类整理 2.查看K8s集群是否安装NFS持久化存储,如果没有则需要安装配置 1).安装NFS服务 2).k8s注册nfs服务 3.修改Prometheus 持久化 4.修改grafana持久化配置 5.修改 promethus和Grafana的Service 端口

    2024年02月08日
    浏览(42)
  • 详解K8s 镜像缓存管理kube-fledged

    本文分享自华为云社区《K8s 镜像缓存管理 kube-fledged 认知》,作者: 山河已无恙。 我们知道  k8s  上的容器调度需要在调度的节点行拉取当前容器的镜像,在一些特殊场景中, 需要 快速启动和/或扩展 的应用程序。例如,由于数据量激增,执行实时数据处理的应用程序需要

    2024年04月15日
    浏览(49)
  • kube-controller-manager和kube-scheduler不能正常启动

     kube-controller-manager-k8s-worker01和kube-scheduler-k8s-worker01没有启动起来 原因: 解决:进入/etc/kubernetes/manifests 编辑 将镜像地址修改为 然后重启kubelet:systemctl restart kubelet.service

    2024年02月07日
    浏览(35)
  • k8s安装promethues,kube-promethues安装法

    目录 k8s安装kube-promethues(0.7版本) 一.检查本地k8s版本,下载对应安装包 二.安装前准备 1.文件分类整理 2.查看K8s集群是否安装NFS持久化存储,如果没有则需要安装配置 1).安装NFS服务 2).k8s注册nfs服务 3.修改Prometheus 持久化 4.修改grafana持久化配置 5.修改 promethus和Grafana的Service 端口

    2024年02月08日
    浏览(76)
  • CKS之k8s安全基准工具:kube-bench

            CIS Kubernetes Benchmark 由互联网安全中心(CIS)社区维护,旨在提供 Kubernetes 的安全配置基线,旨在为互联网环境提供免费的安全防御方案。CIS是一个非营利性组织,其制定的安全基准覆盖了多个领域,包括操作系统、中间件、应用程序等多个层面。         CIS官网:

    2024年04月10日
    浏览(45)
  • k8s安装promethues监控,kube-promethues安装法

    目录 k8s安装kube-promethues(0.7版本) 一.检查本地k8s版本,下载对应安装包 二.安装前准备 1.文件分类整理 2.查看K8s集群是否安装NFS持久化存储,如果没有则需要安装配置 1).安装NFS服务 2).k8s注册nfs服务 3.修改Prometheus 持久化 4.修改grafana持久化配置 5.修改 promethus和Grafana的Service 端口

    2024年02月08日
    浏览(36)
  • 夜莺(Flashcat)V6监控(五):夜莺监控k8s组件(下)---使用kube-state-metrics监控K8s对象

    目录 (一)前言 (二)categraf作为Daemonset的方式去运行监控k8s组件  (1)1.24版本以下的k8s集群部署方法: ①创建autu.yaml绑定权限 ②Daemonset部署categraf采集监控kubelet,kube-proxy ③测试数据是否采集成功  (2)1.24版本以上的k8s集群部署方法: ①创建secret token 绑定sa账号 ③测试认证 ④Daemo

    2024年02月09日
    浏览(44)
  • prometheus监控k8s kube-proxy target down

    解决 修改配置 删除 kube-proxy pod 使之重启应用配置

    2024年02月14日
    浏览(51)
  • K8s安全配置:CIS基准与kube-bench工具

    01、概述 K8s集群往往会因为配置不当导致存在入侵风险,如K8S组件的未授权访问、容器逃逸和横向攻击等。为了保护K8s集群的安全,我们必须仔细检查安全配置。 CIS Kubernetes基准提供了集群安全配置的最佳实践,主要聚焦在两个方面:主节点安全配置和工作节点安全配置。主

    2024年02月14日
    浏览(34)
  • kube-state-metrics暴露k8s中的监控指标

    kube-state-metrics 是一个用于从 Kubernetes 集群中生成各种资源对象状态指标的工具。 通过Deployment等配置完成安装 https://github.com/kubernetes/kube-state-metrics/tree/main/examples/standard 根据官方给定的配置添加至k8s上 注意需要RBAC授权 启动项 要使 kube-state-metrics 暴露更多的节点和 Pod 指标,可

    2024年02月14日
    浏览(43)

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

博客赞助

微信扫一扫打赏

请作者喝杯咖啡吧~博客赞助

支付宝扫一扫领取红包,优惠每天领

二维码1

领取红包

二维码2

领红包