kubernetes - 在 k8s centos7 裸机集群上重新创建部署/Pod 后，服务不允许访问 Pod 的流量：没有到主机的路由

Question

创建了一个裸机多主集群 1.18 版本：

NAME            STATUS   ROLES    AGE   VERSION
master1   Ready    master   9d    v1.18.0
master2   Ready    master   9d    v1.18.0
master3   Ready    master   9d    v1.18.0
worker1   Ready    <none>   9d    v1.18.0
worker2   Ready    <none>   9d    v1.18.0
worker3   Ready    <none>   9d    v1.18.0

kubectl create deploy la --image=linuxacademycontent/kubeserve:v1
kubectl scale deploy la --replica=6
kubectl get pod

curlpod                1/1     Running   0          55m
la-65776bc88f-4kp95    1/1     Running   0          42m
la-65776bc88f-7lb9f    1/1     Running   0          42m
la-65776bc88f-dgzsz    1/1     Running   0          42m
la-65776bc88f-l2hs2    1/1     Running   0          43m
la-65776bc88f-vkcts    1/1     Running   0          42m
la-65776bc88f-wkm2k    1/1     Running   0          42m

kubectl expose deploy la --port=80 --target-port=80
kubectl get svc

kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   75m
la           ClusterIP   10.103.199.187   <none>        80/TCP    46m

kubectl 获取 ep

kubernetes   172.31.16.67:6443                                       78m
la           10.244.1.6:80,10.244.1.7:80,10.244.2.5:80 + 3 more...   49m

我可以通过从 testpod 卷曲来验证一切正常 - curlpod
kubectl exec -it curlpod bash

This is v1 pod la-65776bc88f-dgzsz

问题是如果我现在删除 deploy（或删除所有 pod）并重新创建它们，该服务将不允许流量到新 pod。
这是重现问题的方法。 kubectl 删除部署 la

curlpod                1/1     Running       0          71m
la-65776bc88f-4kp95    1/1     Terminating   0          58m
la-65776bc88f-7lb9f    1/1     Terminating   0          58m
la-65776bc88f-dgzsz    1/1     Terminating   0          58m
la-65776bc88f-l2hs2    1/1     Terminating   0          59m
la-65776bc88f-vkcts    1/1     Terminating   0          58m
la-65776bc88f-wkm2k    1/1     Terminating   0          58m

kubectl create deploy la --image=linuxacademycontent/kubeserve:v1
kubectl scale deploy la --replicas=6
kubectl get pod

NAME                   READY   STATUS    RESTARTS   AGE
curlpod                1/1     Running   0          73m
la-65776bc88f-5q7d8    1/1     Running   0          6s
la-65776bc88f-b69z2    1/1     Running   0          10s
la-65776bc88f-j2t4t    1/1     Running   0          6s
la-65776bc88f-ndxs5    1/1     Running   0          6s
la-65776bc88f-ns2gt    1/1     Running   0          6s
la-65776bc88f-vjp8x    1/1     Running   0          6s

一切看起来都很好。但是，服务不允许任何流量到新的 Pod
kubectl exec -it curlpod bash

root@curlpod:/app# curl la
curl: (7) Failed to connect to la port 80: No route to host
root@curlpod:/app# curl 10.103.199.187   #name resolution seems fine
curl: (7) Failed to connect to la port 80: No route to host

我仍然可以单独到达豆荚。例如 la-65776bc88f-5q7d8 的 IP：10.244.5.13

root@test-pod:/app# curl 10.244.5.13
This is v1 pod la-65776bc88f-5q7d8

所以我认为是服务不会路由流量，但我可以看到服务没有变化，所有新的 pod 都被列为服务的端点。 kubectl 获取 svc

NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP   93m
la           ClusterIP   10.103.199.187   <none>        80/TCP    64m

kubectl 获取 ep

NAME         ENDPOINTS                                               AGE
kubernetes   172.31.16.67:6443                                       93m
la           10.244.5.13,10.244.1.9:80,10.244.2.7:80 + 3 more...   64m

某处阻止了从服务到新 pod 的流量。

现在，如果我们重新启动 svc，问题就会消失！！
kubectl delete svc la
服务“la”已删除

kubectl expose deploy la --port=80 --target-port=80
服务/la暴露

kubectl 获取 svc

kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   108m
la           ClusterIP   10.96.124.200   <none>        80/TCP    71s

kubectl exec -it curlpod bash

This is v1 pod la-65776bc88f-z24qn

尝试了一些替代方案，例如使用 yaml 创建部署、服务，同样的问题。还尝试使用 kubectl scale 缩小和放大。但是我在部署中创建了新的 Pod，从服务到 Pod 的流量总是被阻止。

欢迎任何帮助或建议！

添加法兰绒日志

kubectl logs --namespace kube-system kube-flannel-ds-amd64-tbglf
I0423 15:15:52.311421       1 main.go:518] Determining IP address of default interface
I0423 15:15:52.312644       1 main.go:531] Using interface with name eth0 and address 10.30.193.13
I0423 15:15:52.312669       1 main.go:548] Defaulting external address to interface address (10.30.193.13)
W0423 15:15:52.312689       1 client_config.go:517] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0423 15:15:52.323517       1 kube.go:119] Waiting 10m0s for node controller to sync
I0423 15:15:52.324491       1 kube.go:306] Starting kube subnet manager
I0423 15:15:53.323919       1 kube.go:126] Node controller sync successful
I0423 15:15:53.323976       1 main.go:246] Created subnet manager: Kubernetes Subnet Manager - uk02sybk8smp3
I0423 15:15:53.323983       1 main.go:249] Installing signal handlers
I0423 15:15:53.324084       1 main.go:390] Found network config - Backend type: vxlan
I0423 15:15:53.324176       1 vxlan.go:121] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
I0423 15:15:53.365832       1 main.go:355] Current network or subnet (10.244.0.0/16, 10.244.2.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules
I0423 15:15:53.367815       1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN
I0423 15:15:53.409089       1 iptables.go:167] Deleting iptables rule: -s 0.0.0.0/0 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0423 15:15:53.410118       1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j RETURN
I0423 15:15:53.411017       1 iptables.go:167] Deleting iptables rule: ! -s 0.0.0.0/0 -d 0.0.0.0/0 -j MASQUERADE --random-fully
I0423 15:15:53.412076       1 main.go:305] Setting up masking rules
I0423 15:15:53.412916       1 main.go:313] Changing default FORWARD chain policy to ACCEPT
I0423 15:15:53.412999       1 main.go:321] Wrote subnet file to /run/flannel/subnet.env
I0423 15:15:53.413009       1 main.go:325] Running backend.
I0423 15:15:53.413017       1 main.go:343] Waiting for all goroutines to exit
I0423 15:15:53.413038       1 vxlan_network.go:60] watching for new subnet leases
I0423 15:15:53.510136       1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0423 15:15:53.510158       1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
I0423 15:15:53.510293       1 iptables.go:145] Some iptables rules are missing; deleting and recreating rules
I0423 15:15:53.510301       1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 -j ACCEPT
I0423 15:15:53.511096       1 iptables.go:167] Deleting iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0423 15:15:53.512091       1 iptables.go:167] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.2.0/24 -j RETURN
I0423 15:15:53.512206       1 iptables.go:167] Deleting iptables rule: -d 10.244.0.0/16 -j ACCEPT
I0423 15:15:53.512998       1 iptables.go:167] Deleting iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully
I0423 15:15:53.513970       1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
I0423 15:15:53.609595       1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 -j ACCEPT
I0423 15:15:53.612312       1 iptables.go:155] Adding iptables rule: -d 10.244.0.0/16 -j ACCEPT
I0423 15:15:53.613132       1 iptables.go:155] Adding iptables rule: -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE --random-fully
I0423 15:15:53.710527       1 iptables.go:155] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.2.0/24 -j RETURN
I0423 15:15:53.712938       1 iptables.go:155] Adding iptables rule: ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE --random-fully

kube-proxy 充斥着

E0504 08:25:36.251542       1 proxier.go:1533] Failed to sync endpoint for service: 10.30.218.46:30080/TCP, err: parseIP Error ip=[10 244 5 3 0 0 0 0 0 0 0 0 0 0 0 0]
E0504 08:25:36.251632       1 proxier.go:1950] Failed to list IPVS destinations, error: parseIP Error ip=[10 244 5 3 0 0 0 0 0 0 0 0 0 0 0 0]

标签： kubernetes