【水文】calico-node 启动失败 Init:CrashLoopBackOff

这篇具有很好参考价值的文章主要介绍了【水文】calico-node 启动失败 Init:CrashLoopBackOff。希望对大家有所帮助。如果存在错误或未考虑完全的地方,请大家不吝赐教,您也可以点击"举报违法"按钮提交疑问。

查看日志报错如下 

Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), mount-bpffs (init)
Error from server (BadRequest): container "calico-node" in pod "calico-node-4j7td" is waiting to start: PodInitializing

结果:kube-proxy没启动,每个人的环境不同,需要具体排查看日志。下面是分析过程。

一直没看懂上面第一条啥意思,之前搞calico遇到过各种问题,潜意识觉得calico难搞就无脑搜啊搜,以为是疑难杂症,后来网上看到几个相同第一条报错的,才知道没啥价值,浪费了好多时间。

具体错误如下:

[root@k8s-master01 ~]# kubectl logs -n kube-system calico-node-4j7td
Defaulted container "calico-node" out of: calico-node, upgrade-ipam (init), install-cni (init), mount-bpffs (init)
Error from server (BadRequest): container "calico-node" in pod "calico-node-4j7td" is waiting to start: PodInitializing


[root@k8s-master01 ~]# kubectl get po -A -owide
NAMESPACE     NAME                                       READY   STATUS                  RESTARTS         AGE   IP                NODE           NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-6747f75cdc-pzhhj   1/1     Running                 0                52m   172.27.14.194     k8s-node02     <none>           <none>
kube-system   calico-node-4j7td                          0/1     Init:CrashLoopBackOff   16 (3m58s ago)   43m   192.168.145.161   k8s-master01   <none>           <none>
kube-system   calico-node-hwttj                          1/1     Running                 0                52m   192.168.145.162   k8s-master02   <none>           <none>

【精要】在这个启动失败的节点上,查看容器的日志,如果看不到运行中的容器,就查看exited的容器,如下:

[root@k8s-master01 ~]# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD
[root@k8s-master01 ~]# crictl ps -a
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD
6e2c272bcbd41       8479c67f450d3       4 minutes ago       Exited              install-cni         17                  0b0d02f6ac576       calico-node-4j7td
5c5f54bbda5db       8479c67f450d3       50 minutes ago      Exited              upgrade-ipam        1                   0b0d02f6ac576       calico-node-4j7td
[root@k8s-master01 ~]# crictl logs 6e2c272bcbd41
time="2022-12-01T15:36:27Z" level=info msg="Running as a Kubernetes pod" source="install.go:145"
2022-12-01 15:36:28.196 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/bandwidth"
2022-12-01 15:36:28.197 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/bandwidth
2022-12-01 15:36:28.297 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/calico"
2022-12-01 15:36:28.297 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/calico
2022-12-01 15:36:28.373 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/calico-ipam"
2022-12-01 15:36:28.373 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/calico-ipam
2022-12-01 15:36:28.376 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/flannel"
2022-12-01 15:36:28.376 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/flannel
2022-12-01 15:36:28.381 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/host-local"
2022-12-01 15:36:28.381 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/host-local
2022-12-01 15:36:28.447 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/install"
2022-12-01 15:36:28.447 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/install
2022-12-01 15:36:28.451 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/loopback"
2022-12-01 15:36:28.451 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/loopback
2022-12-01 15:36:28.455 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/portmap"
2022-12-01 15:36:28.455 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/portmap
2022-12-01 15:36:28.459 [INFO][1] cni-installer/<nil> <nil>: File is already up to date, skipping file="/host/opt/cni/bin/tuning"
2022-12-01 15:36:28.459 [INFO][1] cni-installer/<nil> <nil>: Installed /host/opt/cni/bin/tuning
2022-12-01 15:36:28.459 [INFO][1] cni-installer/<nil> <nil>: Wrote Calico CNI binaries to /host/opt/cni/bin

2022-12-01 15:36:28.494 [INFO][1] cni-installer/<nil> <nil>: CNI plugin version: v3.25.0-0.dev-519-g2fee4ee0153d

2022-12-01 15:36:28.494 [INFO][1] cni-installer/<nil> <nil>: /host/secondary-bin-dir is not writeable, skipping
W1201 15:36:28.494754       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2022-12-01 15:36:49.499 [ERROR][1] cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=Post "https://10.96.0.1:443/api/v1/namespaces/kube-system/serviceaccounts/calico-node/token": dial tcp 10.96.0.1:443: connect: connection refused
2022-12-01 15:36:49.499 [FATAL][1] cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=Post "https://10.96.0.1:443/api/v1/namespaces/kube-system/serviceaccounts/calico-node/token": dial tcp 10.96.0.1:443: connect: connection refused
[root@k8s-master01 ~]#
[root@k8s-master01 ~]#
[root@k8s-master01 ~]#
[root@k8s-master01 ~]#
[root@k8s-master01 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d16h
[root@k8s-master01 ~]# kubectl get svc  -A
NAMESPACE     NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
default       kubernetes     ClusterIP   10.96.0.1       <none>        443/TCP    2d16h
kube-system   calico-typha   ClusterIP   10.107.84.250   <none>        5473/TCP   64m
[root@k8s-master01 ~]#
[root@k8s-master01 ~]#
[root@k8s-master01 ~]# telnet 10.96.0.1 443
Trying 10.96.0.1...
^C
[root@k8s-master01 ~]# ping 10.96.0.1
PING 10.96.0.1 (10.96.0.1) 56(84) bytes of data.
^C
--- 10.96.0.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

上面发现10.96.0.1不通,去其他节点查看是通的

[root@k8s-node02 ~]#
[root@k8s-node02 ~]# telnet 10.96.0.1 443
Trying 10.96.0.1...
Connected to 10.96.0.1.
Escape character is '^]'.
^CConnection closed by foreign host.
[root@k8s-node02 ~]# ping 10.96.0.1
PING 10.96.0.1 (10.96.0.1) 56(84) bytes of data.
64 bytes from 10.96.0.1: icmp_seq=1 ttl=64 time=0.061 ms
64 bytes from 10.96.0.1: icmp_seq=2 ttl=64 time=0.073 ms
^C
--- 10.96.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1057ms
rtt min/avg/max/mdev = 0.061/0.067/0.073/0.006 ms
[root@k8s-node02 ~]# ping 10.96.0.1
PING 10.96.0.1 (10.96.0.1) 56(84) bytes of data.
64 bytes from 10.96.0.1: icmp_seq=1 ttl=64 time=0.056 ms


64 bytes from 10.96.0.1: icmp_seq=2 ttl=64 time=0.067 ms
^C
--- 10.96.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1003ms
rtt min/avg/max/mdev = 0.056/0.061/0.067/0.009 ms

查看问题节点的代理服务kube-proxy,果然,这个居然是disable,郁闷,启动后,delete掉失败的pod,瞬间好了

[root@k8s-master01 ~]# systemctl status kube-proxy
● kube-proxy.service - Kubernetes Kube Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: https://github.com/kubernetes/kubernetes
[root@k8s-master01 ~]# systemctl enable --now kube-proxy
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
[root@k8s-master01 ~]# systemctl status kube-proxy
● kube-proxy.service - Kubernetes Kube Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-12-01 23:43:25 CST; 2s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 20930 (kube-proxy)
    Tasks: 7
   Memory: 52.7M
   CGroup: /system.slice/kube-proxy.service
           └─20930 /usr/local/bin/kube-proxy --config=/etc/kubernetes/kube-proxy.yaml --v=2



[root@k8s-master01 ~]# kubectl get po -A -owide -w
NAMESPACE     NAME                                       READY   STATUS                  RESTARTS         AGE   IP                NODE           NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-6747f75cdc-pzhhj   1/1     Running                 0                68m   172.27.14.194     k8s-node02     <none>           <none>
kube-system   calico-node-4j7td                          0/1     Init:CrashLoopBackOff   19 (3m27s ago)   58m   192.168.145.161   k8s-master01   <none>           <none>
kube-system   calico-node-hwttj                          1/1     Running                 0                68m   192.168.145.162   k8s-master02   <none>           <none>
kube-system   calico-node-rjbz8                          1/1     Running                 0                68m   192.168.145.163   k8s-master03   <none>           <none>
kube-system   calico-node-rmjqj                          1/1     Running                 1 (55m ago)      68m   192.168.145.165   k8s-node02     <none>           <none>
kube-system   calico-node-vd7w2                          1/1     Running                 0                68m   192.168.145.164   k8s-node01     <none>           <none>
kube-system   calico-typha-6cdc4b4fbc-sb85z              1/1     Running                 0                68m   192.168.145.164   k8s-node01     <none>           <none>


^C[root@k8s-master01 ~]# kubectl delete po -n kube-system calico-node-4j7td
pod "calico-node-4j7td" deleted
[root@k8s-master01 ~]# kubectl get po -A -owide -w
NAMESPACE     NAME                                       READY   STATUS     RESTARTS      AGE   IP                NODE           NOMINATED NODE   READINESS GATES
kube-system   calico-kube-controllers-6747f75cdc-pzhhj   1/1     Running    0             68m   172.27.14.194     k8s-node02     <none>           <none>
kube-system   calico-node-hwttj                          1/1     Running    0             68m   192.168.145.162   k8s-master02   <none>           <none>
kube-system   calico-node-jbvlc                          0/1     Init:1/3   0             2s    192.168.145.161   k8s-master01   <none>           <none>
kube-system   calico-node-rjbz8                          1/1     Running    0             68m   192.168.145.163   k8s-master03   <none>           <none>
kube-system   calico-node-rmjqj                          1/1     Running    1 (55m ago)   68m   192.168.145.165   k8s-node02     <none>           <none>
kube-system   calico-node-vd7w2                          1/1     Running    0             68m   192.168.145.164   k8s-node01     <none>           <none>
kube-system   calico-typha-6cdc4b4fbc-sb85z              1/1     Running    0             68m   192.168.145.164   k8s-node01     <none>           <none>
kube-system   calico-node-jbvlc                          0/1     Init:1/3   0             2s    192.168.145.161   k8s-master01   <none>           <none>
kube-system   calico-node-jbvlc                          0/1     Init:2/3   0             3s    192.168.145.161   k8s-master01   <none>           <none>
kube-system   calico-node-jbvlc                          0/1     PodInitializing   0             4s    192.168.145.161   k8s-master01   <none>           <none>
kube-system   calico-node-jbvlc                          0/1     Running           0             5s    192.168.145.161   k8s-master01   <none>           <none>
^C[root@k8s-master01 ~]#

世上无难事,只怕有心人还不够,要有脑子,不要浪费不该浪费的时间。文章来源地址https://www.toymoban.com/news/detail-514004.html

到了这里,关于【水文】calico-node 启动失败 Init:CrashLoopBackOff的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处: 如若内容造成侵权/违法违规/事实不符,请点击违法举报进行投诉反馈,一经查实,立即删除!

领支付宝红包 赞助服务器费用

相关文章

  • k8s部署Calico网络插件出现Init:ImagePullBackOff

    配置环境: CentOS 7.9 docker-ce-19.03.15 kubeadm-1.22.17-0 kubelet-1.22.17-0 kubectl-1.22.17-0 配置过程略。。。 在部署单Master时,想要其他的node节点加入到集群中来,然后就显示其中一个node节点的准备状态是NotReady  此时这里我们用到的网络插件:Calico kubectl apply -f  https://gitee.com/leedon21/k

    2024年02月16日
    浏览(40)
  • K8s的Pod出现Init:ImagePullBackOff问题的解决(以calico为例)

    对于这类问题的解决思路应该都差不多,本文以calico插件安装为例,发现有个Pod的镜像没有pull成功 从上图发现是docker拉取\\\"calico/cni:v3.15.1\\\"失败,那么我们手动拉取一下 Note :我们需要知道这个pod是部署在哪个node上,然后才能在具体的Node上拉取镜像 发现是在k8snode1主机上,那

    2024年02月11日
    浏览(48)
  • Pod无法正常启动,出现CrashLoopBackOff状态

    Pod无法正常启动,出现CrashLoopBackOff状态。这个状态表示Kubernetes已经尝试了多次重新启动Pod,但是每次都失败了。 这种情况的原因有很多,以下是一些常见的原因以及相应的解决方法: 容器镜像拉取失败:可能是由于网络问题导致容器镜像无法下载。可以尝试使用 kubectl de

    2024年02月11日
    浏览(39)
  • RabbitMQ启动失败 ERROR: node with name "rabbit" already running on "xxx"

    首先,你得先确认的你得Erlang的版本和RabbitMQ的版本是否能相互兼容,如果不能,请先下载对应的Erlang版本或者RabbitMQ版本。 1. 在Java中使用了RabbitMQ发现不停的尝试连接,却连不上的问题 先在cmd下输入erl,看一下Erlang的环境变量配没有 没有上图这个结果的,先配置对应的Er

    2024年02月12日
    浏览(53)
  • 解决ES集群启动失败 skipping cluster bootstrapping as local node does not match bootstrap requirements:[xxx]

    异常信息 这个错误信息网上有很多解决方案,但都没说明一点,就是由于账号密码配置而导致的连接失败 正常根据网上资源和官方文档介绍后,我们会进行以下配置 原配置信息 我们目前集群信息是一主一从,没办法白嫖的阿里云机子才 2g 只能这样子配置 集群名称: es-kk

    2024年02月04日
    浏览(51)
  • 关于k8s 安装Dashboard recommended.yaml下载失败以及calico.yaml 下载文件

    输入地址: https://www.ipaddress.com/ 查找 查询结果: 然后配置 linux 文件 vim /etc/hosts 然后再执行: calico.yaml 下载地址:https://docs.projectcalico.org/v3.8/manifests/calico.yaml

    2024年02月16日
    浏览(53)
  • Calico 报错 calico/node is not ready: BIRD is not ready: BGP not established with(导致 K8S 集群网络无法使用)

    错误 今天不知道怎么回事,一台机器的calico-node报错,也就是无法初始化正常 解决办法         移除这台主机多余的docker网卡和calico         然后从重新删除这个错误pod的,就会恢复正常         造成问题的原因是,Calico使用了错误的网桥导致网络无法连通,所以我

    2024年02月07日
    浏览(43)
  • 解决公网下,k8s calico master节点无法访问node节点创建的pod

    目的:解决pod部署成功后,只能在node节点访问,而master节点无法访问 原因:集群搭建时,没有配置公网进行kubectl操作,从而导致系统默认node节点,使用内网IP加入k8s集群!如下: 解决方案:围绕公网IP进行搭建即可,其实就是在传统的搭建步骤下,给master节点和node节点添加

    2024年02月03日
    浏览(50)
  • git submodule update --init 失败解决办法

    我们在github上寻找开源项目的源码,clone下来研究的时候,由于项目需要多个开源项目的支持,所有还需要二次clone子工程的源码支持,比如最近在使用: grpc 在clone子模块的时候会出现失败或无法访问的情况,其原因众所周知无非就是有些东西需要科学上网 找到项目的.gitmodul

    2024年02月16日
    浏览(40)
  • [Spring5.3.2] Servlet[springmvc]的Servlet.init()引发异常, 解析类文件失败

    问题表现: 图中提到的问题: 例外情况 javax.servlet.ServletException: Servlet[springmvc]的Servlet.init()引发异常 根本原因 org.springframework.beans.factory.BeanDefinitionStoreException: Failed to read candidate component class: file [D:apache-tomcat-9.0.69webappsch2_2WEB-INFclassescontrollerIndexController.class]; nested excep

    2024年02月02日
    浏览(33)

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

博客赞助

微信扫一扫打赏

请作者喝杯咖啡吧~博客赞助

支付宝扫一扫领取红包,优惠每天领

二维码1

领取红包

二维码2

领红包