Kubernetes部署ingress

环境

• RHEL 9.3
• Docker Community 24.0.7
• minikube v1.32.0

部署ingress

可以访问 registry.k8s.io 的环境

通过minikube启用 ingress ：

minikube addons enable ingress

注：接下来在不能访问 registry.k8s.io 的环境里，用该环境作为“标准环境”，以便做对比。

不能访问 registry.k8s.io 的环境

通过minikube启用 ingress 会报错：

$ minikube addons enable ingress
💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.9.4
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
🔎  Verifying ingress addon...

❌  Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: context deadline exceeded]

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│    Please also attach the following file to the GitHub issue:                             │
│    - /tmp/minikube_addons_bbcace35d8e0350ae758b20ff8fff60714ee7a80_0.log                  │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

ingress使用的是 ingress-nginx namespace：

$ kubectl get namespace
NAME                   STATUS   AGE
default                Active   42h
ingress-nginx          Active   31h
kube-node-lease        Active   42h
kube-public            Active   42h
kube-system            Active   42h
kubernetes-dashboard   Active   41h

查看 ingress-nginx namespace：

$ kubectl get all -n ingress-nginx
NAME                                            READY   STATUS              RESTARTS   AGE
pod/ingress-nginx-admission-create-274rs        0/1     ImagePullBackOff    0          51s
pod/ingress-nginx-admission-patch-tg887         0/1     ContainerCreating   0          51s
pod/ingress-nginx-controller-7c6974c4d8-lq5q4   0/1     ContainerCreating   0          51s

NAME                                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             NodePort    10.111.104.1    <none>        80:31322/TCP,443:31156/TCP   51s
service/ingress-nginx-controller-admission   ClusterIP   10.100.55.232   <none>        443/TCP                      51s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   0/1     1            0           51s

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-7c6974c4d8   1         1         0       51s

NAME                                       COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   0/1           51s        51s
job.batch/ingress-nginx-admission-patch    0/1           51s        51s

查看deployment：

$ kubectl describe deployment ingress-nginx-controller -n ingress-nginx
......
    Image:       registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
......
  Volumes:
   webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
......
NewReplicaSet:   ingress-nginx-controller-7c6974c4d8 (1/1 replicas created)
Events:
  Type    Reason             Age    From                   Message
  ----    ------             ----   ----                   -------
  Normal  ScalingReplicaSet  3m22s  deployment-controller  Scaled up replica set ingress-nginx-controller-7c6974c4d8 to 1

$ kubectl describe replicaset ingress-nginx-controller-7c6974c4d8 -n ingress-nginx
......
Controlled By:  Deployment/ingress-nginx-controller
......
  Containers:
   controller:
    Image:       registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
......
  Volumes:
   webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
Events:
  Type    Reason            Age   From                   Message
  ----    ------            ----  ----                   -------
  Normal  SuccessfulCreate  11m   replicaset-controller  Created pod: ingress-nginx-controller-7c6974c4d8-lq5q4

$ kubectl describe pod ingress-nginx-controller-7c6974c4d8-lq5q4 -n ingress-nginx
......
Controlled By:    ReplicaSet/ingress-nginx-controller-7c6974c4d8
Containers:
  controller:
    Container ID:  
    Image:         registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
......
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
......
Events:
  Type     Reason       Age                 From               Message
  ----     ------       ----                ----               -------
  Normal   Scheduled    13m                 default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-controller-7c6974c4d8-lq5q4 to minikube
  Warning  FailedMount  65s (x14 over 13m)  kubelet            MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found

查看job：

$ kubectl describe job ingress-nginx-admission-create -n ingress-nginx
......
  Containers:
   create:
    Image:      registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
......
Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  22m   job-controller  Created pod: ingress-nginx-admission-create-274rs

$ kubectl describe pod ingress-nginx-admission-create-274rs -n ingress-nginx
......
Controlled By:  Job/ingress-nginx-admission-create
Containers:
  create:
    Container ID:  
    Image:         registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
......
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  16m                 default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-admission-create-274rs to minikube
  Normal   Pulling    12m (x4 over 16m)   kubelet            Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"
  Warning  Failed     12m (x4 over 15m)   kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": dial tcp 108.177.97.82:443: i/o timeout
  Warning  Failed     12m (x4 over 15m)   kubelet            Error: ErrImagePull
  Warning  Failed     11m (x6 over 15m)   kubelet            Error: ImagePullBackOff
  Normal   BackOff    56s (x44 over 15m)  kubelet            Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"

$ kubectl describe job ingress-nginx-admission-patch -n ingress-nginx
......
  Containers:
   patch:
    Image:      registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
......
Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  25m   job-controller  Created pod: ingress-nginx-admission-patch-tg887

$ kubectl describe pod ingress-nginx-admission-patch-tg887 -n ingress-nginx
......
Controlled By:  Job/ingress-nginx-admission-patch
Containers:
  patch:
    Container ID:  
    Image:         registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
......
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  17m                   default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-admission-patch-tg887 to minikube
  Normal   Pulling    14m (x4 over 17m)     kubelet            Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"
  Warning  Failed     13m (x4 over 16m)     kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": dial tcp 108.177.97.82:443: i/o timeout
  Warning  Failed     13m (x4 over 16m)     kubelet            Error: ErrImagePull
  Warning  Failed     12m (x7 over 16m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    7m43s (x23 over 16m)  kubelet            Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"
  Warning  Failed     2m32s                 kubelet            Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": dial tcp 142.251.8.82:443: i/o timeout

所以，归根到底，还是 registry.k8s.io 的网络问题。从 minikube addons enable ingress 的输出可知，使用到的image是：

• registry.k8s.io/ingress-nginx/controller:v1.9.4
• registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0

注：从上述 kubectl describe xxx 命令也可以看出，是这两个image pull不下来。

所以，接下来我们用手工方式来部署ingress。

手工部署

在可以访问 registry.k8s.io 的地方，找到上述两个image，pull下来，tag一下，push到DockerHub自己的repository里。

比如：我的image是：

• kaiding1/controller:v1.9.4
• kaiding1/kube-webhook-certgen:v20231011-8b53cabe0

打开浏览器，访问 https://github.com/kubernetes/ingress-nginx ，其中“Supported Versions table”如下：

查看Kubernetes版本：

$ kubectl version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

由对照表可知，k8s v1.28.3支持的Ingress-Nginx版本有v1.9.0到v1.9.5。我们就用v1.9.4，和 registry.k8s.io/ingress-nginx/controller:v1.9.4 保持一致（也和“标准环境”保持一致）。

把 https://github.com/kubernetes/ingress-nginx/blob/controller-v1.9.4/deploy/static/provider/kind/deploy.yaml 下载到本地。

注：有kind、cloud、baremetal等多个版本，我比较了一下，kind里的版本最接近“标准环境”。

然后修改 deploy.yaml ：

• 把：

image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3

替换为：

image: kaiding1/controller:v1.9.4

• 把：

image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80

替换为：

image: kaiding1/kube-webhook-certgen:v20231011-8b53cabe0

注意有两处 kube-webhook-certgen 要替换。

注：删掉了 @sha256:xxxxx ，因为跟原来的digest不一样，如果想加上，需要使用新的digest。

部署该yaml文件：

$ kubectl apply -f deploy.yaml
namespace/ingress-nginx unchanged
serviceaccount/ingress-nginx configured
serviceaccount/ingress-nginx-admission configured
role.rbac.authorization.k8s.io/ingress-nginx configured
role.rbac.authorization.k8s.io/ingress-nginx-admission configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission configured
rolebinding.rbac.authorization.k8s.io/ingress-nginx configured
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission configured
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx configured
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission configured
configmap/ingress-nginx-controller configured
service/ingress-nginx-controller configured
service/ingress-nginx-controller-admission configured
deployment.apps/ingress-nginx-controller configured
ingressclass.networking.k8s.io/nginx configured
networkpolicy.networking.k8s.io/ingress-nginx-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/component\":\"admission-webhook\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"1.9.4\"},\"name\":\"ingress-nginx-admission-create\",\"namespace\":\"ingress-nginx\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"admission-webhook\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"1.9.4\"},\"name\":\"ingress-nginx-admission-create\"},\"spec\":{\"containers\":[{\"args\":[\"create\",\"--host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc\",\"--namespace=$(POD_NAMESPACE)\",\"--secret-name=ingress-nginx-admission\"],\"env\":[{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}}],\"image\":\"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"create\",\"securityContext\":{\"allowPrivilegeEscalation\":false}}],\"nodeSelector\":{\"kubernetes.io/os\":\"linux\"},\"restartPolicy\":\"OnFailure\",\"securityContext\":{\"fsGroup\":2000,\"runAsNonRoot\":true,\"runAsUser\":2000},\"serviceAccountName\":\"ingress-nginx-admission\"}}}}\n"},"labels":{"app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.4"}},"spec":{"template":{"metadata":{"labels":{"app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.4"}},"spec":{"$setElementOrder/containers":[{"name":"create"}],"containers":[{"image":"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0","name":"create"}],"nodeSelector":{"minikube.k8s.io/primary":null},"securityContext":{"fsGroup":2000}}}}}
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "ingress-nginx-admission-create", Namespace: "ingress-nginx"
for: "deploy.yaml": error when patching "deploy.yaml": Job.batch "ingress-nginx-admission-create" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"ingress-nginx-admission-create", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app.kubernetes.io/component":"admission-webhook", "app.kubernetes.io/instance":"ingress-nginx", "app.kubernetes.io/name":"ingress-nginx", "app.kubernetes.io/part-of":"ingress-nginx", "app.kubernetes.io/version":"1.9.4", "batch.kubernetes.io/controller-uid":"4dc1f43f-04d8-4e43-9469-addcd7a96f8c", "batch.kubernetes.io/job-name":"ingress-nginx-admission-create", "controller-uid":"4dc1f43f-04d8-4e43-9469-addcd7a96f8c", "job-name":"ingress-nginx-admission-create"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"create", Image:"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0", Command:[]string(nil), Args:[]string{"create", "--host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc", "--namespace=$(POD_NAMESPACE)", "--secret-name=ingress-nginx-admission"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"POD_NAMESPACE", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00b2e2720)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(0xc00b40c2a0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc00d6fdd40), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"kubernetes.io/os":"linux"}, ServiceAccountName:"ingress-nginx-admission", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc00b52e480), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/component\":\"admission-webhook\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"1.9.4\"},\"name\":\"ingress-nginx-admission-patch\",\"namespace\":\"ingress-nginx\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"admission-webhook\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"1.9.4\"},\"name\":\"ingress-nginx-admission-patch\"},\"spec\":{\"containers\":[{\"args\":[\"patch\",\"--webhook-name=ingress-nginx-admission\",\"--namespace=$(POD_NAMESPACE)\",\"--patch-mutating=false\",\"--secret-name=ingress-nginx-admission\",\"--patch-failure-policy=Fail\"],\"env\":[{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}}],\"image\":\"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"patch\",\"securityContext\":{\"allowPrivilegeEscalation\":false}}],\"nodeSelector\":{\"kubernetes.io/os\":\"linux\"},\"restartPolicy\":\"OnFailure\",\"securityContext\":{\"fsGroup\":2000,\"runAsNonRoot\":true,\"runAsUser\":2000},\"serviceAccountName\":\"ingress-nginx-admission\"}}}}\n"},"labels":{"app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.4"}},"spec":{"template":{"metadata":{"labels":{"app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.4"}},"spec":{"$setElementOrder/containers":[{"name":"patch"}],"containers":[{"image":"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0","name":"patch"}],"nodeSelector":{"minikube.k8s.io/primary":null},"securityContext":{"fsGroup":2000}}}}}
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "ingress-nginx-admission-patch", Namespace: "ingress-nginx"
for: "deploy.yaml": error when patching "deploy.yaml": Job.batch "ingress-nginx-admission-patch" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"ingress-nginx-admission-patch", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app.kubernetes.io/component":"admission-webhook", "app.kubernetes.io/instance":"ingress-nginx", "app.kubernetes.io/name":"ingress-nginx", "app.kubernetes.io/part-of":"ingress-nginx", "app.kubernetes.io/version":"1.9.4", "batch.kubernetes.io/controller-uid":"1033bcb2-b2f4-4a6a-b43c-15ad9351d6ec", "batch.kubernetes.io/job-name":"ingress-nginx-admission-patch", "controller-uid":"1033bcb2-b2f4-4a6a-b43c-15ad9351d6ec", "job-name":"ingress-nginx-admission-patch"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"patch", Image:"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0", Command:[]string(nil), Args:[]string{"patch", "--webhook-name=ingress-nginx-admission", "--namespace=$(POD_NAMESPACE)", "--patch-mutating=false", "--secret-name=ingress-nginx-admission", "--patch-failure-policy=Fail"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"POD_NAMESPACE", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00bb53f80)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(0xc00bb5d440), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc00bb82c00), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"kubernetes.io/os":"linux"}, ServiceAccountName:"ingress-nginx-admission", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc00bb75d40), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable

有些配置跟原来已有的配置冲突，为了简单以前，把原来的namespace整个删掉：

kubectl delete namespace ingress-nginx

所有的deployment、pod都资源都没了，然后再重新部署：

$ kubectl apply -f deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx unchanged
networkpolicy.networking.k8s.io/ingress-nginx-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured

再次查看资源：

$ kubectl get all -n ingress-nginx
NAME                                            READY   STATUS      RESTARTS   AGE
pod/ingress-nginx-admission-create-sbrf8        0/1     Completed   0          40s
pod/ingress-nginx-admission-patch-ftgxj         0/1     Completed   1          40s
pod/ingress-nginx-controller-6d5bdfb648-wkrdr   0/1     Pending     0          40s

NAME                                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             NodePort    10.103.85.240   <none>        80:30507/TCP,443:32553/TCP   40s
service/ingress-nginx-controller-admission   ClusterIP   10.96.189.167   <none>        443/TCP                      40s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   0/1     1            0           40s

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-6d5bdfb648   1         1         0       40s

NAME                                       COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   1/1           4s         40s
job.batch/ingress-nginx-admission-patch    1/1           5s         40s

可见， /ingress-nginx-controller 的pod没有启起来，处于“Pending”状态。

查看该pod：

$ kubectl describe pod ingress-nginx-controller-6d5bdfb648-wkrdr -n ingress-nginx
......
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  97s   default-scheduler  0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

和“标准环境”的 ingress-nginx-controller deployment做对比，发现需要注释掉这一行：

apiVersion: apps/v1
kind: Deployment
......
      nodeSelector:
        # ingress-ready: "true" # 把这一行注释掉

还有一些别的不同之处，但是貌似不影响，只有这一行是有影响的。

然后再次删掉 ingress-nginx namespace重建：

kubectl delete namespace ingress-nginx

kubectl apply -f deploy.yaml

查看资源：

$ kubectl get all -n ingress-nginx
NAME                                            READY   STATUS      RESTARTS   AGE
pod/ingress-nginx-admission-create-pk7jj        0/1     Completed   0          13s
pod/ingress-nginx-admission-patch-2twvg         0/1     Completed   0          13s
pod/ingress-nginx-controller-68db9c8cbf-zvsmr   1/1     Running     0          13s

NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/ingress-nginx-controller             NodePort    10.101.105.112   <none>        80:31854/TCP,443:31461/TCP   13s
service/ingress-nginx-controller-admission   ClusterIP   10.98.77.135     <none>        443/TCP                      13s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ingress-nginx-controller   1/1     1            1           13s

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/ingress-nginx-controller-68db9c8cbf   1         1         1       13s

NAME                                       COMPLETIONS   DURATION   AGE
job.batch/ingress-nginx-admission-create   1/1           4s         13s
job.batch/ingress-nginx-admission-patch    1/1           4s         13s

现在所有资源都OK了。

总结

deploy.yaml 需要修改之处：

• image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3 ：替换为可访问的image
• image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80 ：替换为可访问的image，注意有两处
• ingress-ready: "true" ：注释掉这一行

测试ingress

创建 ingress-example.yaml 文件如下：

kind: Pod
apiVersion: v1
metadata:
  name: foo-app
  labels:
    app: foo
spec:
  containers:
    - name: foo-app
      image: 'kicbase/echo-server:1.0'
---
kind: Service
apiVersion: v1
metadata:
  name: foo-service
spec:
  selector:
    app: foo
  ports:
    - port: 8080
---
kind: Pod
apiVersion: v1
metadata:
  name: bar-app
  labels:
    app: bar
spec:
  containers:
    - name: bar-app
      image: 'kicbase/echo-server:1.0'
---
kind: Service
apiVersion: v1
metadata:
  name: bar-service
spec:
  selector:
    app: bar
  ports:
    - port: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
spec:
  rules:
    - http:
        paths:
          - pathType: Prefix
            path: /foo
            backend:
              service:
                name: foo-service
                port:
                  number: 8080
          - pathType: Prefix
            path: /bar
            backend:
              service:
                name: bar-service
                port:
                  number: 8080
---

部署：

$ kubectl apply -f ingress-example.yaml
pod/foo-app created
service/foo-service created
pod/bar-app created
service/bar-service created
ingress.networking.k8s.io/example-ingress created

（注意：部署完以后查看一下这些pod，有时会遇到pull不下来的情况，报错是“You have reached your pull rate limit”。解决方法是 docker login 。但是今天好像login了也不行……我没有细看这个问题，直接用老办法，把它pull下来然后push到个人的repository了。）

查看ingress：

$ kubectl get ingress
NAME              CLASS    HOSTS   ADDRESS     PORTS   AGE
example-ingress   <none>   *       localhost   80      39s

但是，不能用localhost作为host：

$ curl http://localhost/foo
curl: (7) Failed to connect to localhost port 80: Connection refused

不知道是不是我哪里配置的有问题。

而要用 192.168.49.2 （注： 192.168.49.2 是minikube的internal IP）：

$ curl http://192.168.49.2/foo
Request served by foo-app

HTTP/1.1 GET /foo

Host: 192.168.49.2
Accept: */*
User-Agent: curl/7.76.1
X-Forwarded-For: 192.168.49.1
X-Forwarded-Host: 192.168.49.2
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Scheme: http
X-Real-Ip: 192.168.49.1
X-Request-Id: 47a53f8a53335fd2e412c08d52918966
X-Scheme: http

同理：

$ curl http://192.168.49.2/bar
Request served by bar-app

HTTP/1.1 GET /bar

Host: 192.168.49.2
Accept: */*
User-Agent: curl/7.76.1
X-Forwarded-For: 192.168.49.1
X-Forwarded-Host: 192.168.49.2
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Scheme: http
X-Real-Ip: 192.168.49.1
X-Request-Id: c471ed7ba12a154cafe052107d553480
X-Scheme: http

查看 exmaple-ingress ，如下：文章来源地址https://www.toymoban.com/news/detail-808284.html

$ kubectl describe ingress example-ingress
Name:             example-ingress
Labels:           <none>
Namespace:        default
Address:          localhost
Ingress Class:    <none>
Default backend:  <default>
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /foo   foo-service:8080 (10.244.0.34:8080)
              /bar   bar-service:8080 (10.244.0.35:8080)
Annotations:  <none>
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    52m (x2 over 53m)  nginx-ingress-controller  Scheduled for sync

参考

• https://minikube.sigs.k8s.io/docs/start
• https://blog.csdn.net/alwaysbefine/article/details/129201448

到了这里，关于Kubernetes部署ingress的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！