kubernetes - Pod 陷入关闭状态
问题描述
我可以看到这条消息：节点正在关闭，在 pod 描述中逐出 pod，这仅发生在具有特定容忍度的 pod 上，可抢占节点池上的节点选择器。
我们为 pod 添加了容忍度，并创建了具有不同 taints（可抢占、不可抢占）的不同节点池，以隔离集群上的可抢占和不可抢占 pod。
没有污点的集群工作正常。
带有污点的集群存在 pod 陷入关闭状态的问题（仅部署在抢占式节点池上的 pod）
这是吊舱说明
Namespace: XXXXXX
Priority: 0
Node: gke-cluster-reliable-preemptible-node-XXXXXX
Start Time: Tue, 10 Aug 2021 16:44:30 +0530
Labels: app=XXXX
pod-template-hash=XXXX
release=XXXX
repo=XXX
Annotations: randVersion: a200a
Status: Failed
Reason: Shutdown
Message: Node is shutting, evicting pods
IP:
IPs: <none>
Controlled By: ReplicaSet/career-assessor-be-8467d6c885
Containers:
career-assessor-be:
Image: XXXXXX
Port: 8001/TCP
key: CLOUD_SQL_CONNECTION_NAME
Host Port: 0/TCP
Command:
/bin/sh
-c
Args:
XXXXX
Limits:
cpu: 3200m
memory: 2400Mi
Requests:
cpu: 1600m
memory: 1800Mi
Environment Variables from:
careerassessor-config ConfigMap Optional: false
Environment:
LOG_TO_CONSOLE: 1
INACTIVITY_PERIOD:
USER_EMAIL: jyostna@springboard.com
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xpdwd (ro)
cloudsql-proxy:
Image: gcr.io/cloudsql-docker/gce-proxy:1.17
Port: <none>
Host Port: <none>
Command:
/cloud_sql_proxy
-instances=$(CLOUD_SQL_CONNECTION_NAME)=tcp:0.0.0.0:3306
-credential_file=/secrets/cloudsql/cloudsql-instance-credentials.json
-term_timeout=$(CLOUD_SQL_CONNECTION_TIMEOUT)s
Limits:
cpu: 100m
memory: 50Mi
Requests:
cpu: 20m
memory: 20Mi
Environment:
CLOUD_SQL_CONNECTION_NAME: <set to the key 'CLOUD_SQL_CONNECTION_NAME' of config map 'careerassessor-config'> Optional: false
CLOUD_SQL_CONNECTION_TIMEOUT: <set to the key 'CLOUD_SQL_CONNECTION_TIMEOUT' of config map 'careerassessor-config'> Optional: false
Mounts:
/secrets/cloudsql from careerassessor-cloudsql-instance-credentials (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xpdwd (ro)
Volumes:
careerassessor-cloudsql-instance-credentials:
Type: Secret (a volume populated by a Secret)
SecretName: XXXXX
Optional: false
default-token-xpdwd:
Type: Secret (a volume populated by a Secret)
SecretName: XXX
Optional: false
QoS Class: Burstable
Node-Selectors: non-preemptible=false
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
non-preemptible=false:NoSchedule
Events: <none>
这是 pod 的 yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
randVersion: a200a
creationTimestamp: "2021-08-10T10:59:29Z"
generateName: xxx
labels:
app: xxx
pod-template-hash: 8467d6c885
release: xxxx
repo: xxx
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:randVersion: {}
f:generateName: {}
f:labels:
.: {}
f:app: {}
f:pod-template-hash: {}
f:release: {}
f:repo: {}
f:ownerReferences:
.: {}
k:{"uid":"674b9e8e-420e-44e7-9601-871be01a9fcb"}:
.: {}
f:apiVersion: {}
f:blockOwnerDeletion: {}
f:controller: {}
f:kind: {}
f:name: {}
f:uid: {}
f:spec:
f:containers:
k:{"name":"career-assessor-be"}:
.: {}
f:args: {}
f:command: {}
f:env:
.: {}
k:{"name":"INACTIVITY_PERIOD"}:
.: {}
f:name: {}
k:{"name":"LOG_TO_CONSOLE"}:
.: {}
f:name: {}
f:value: {}
k:{"name":"USER_EMAIL"}:
.: {}
f:name: {}
f:value: {}
f:envFrom: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:ports:
.: {}
k:{"containerPort":8001,"protocol":"TCP"}:
.: {}
f:containerPort: {}
f:name: {}
f:protocol: {}
f:resources:
.: {}
f:limits:
.: {}
f:cpu: {}
f:memory: {}
f:requests:
.: {}
f:cpu: {}
f:memory: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
k:{"name":"cloudsql-proxy"}:
.: {}
f:command: {}
f:env:
.: {}
k:{"name":"CLOUD_SQL_CONNECTION_NAME"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:configMapKeyRef:
.: {}
f:key: {}
f:name: {}
k:{"name":"CLOUD_SQL_CONNECTION_TIMEOUT"}:
.: {}
f:name: {}
f:valueFrom:
.: {}
f:configMapKeyRef:
.: {}
f:key: {}
f:name: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources:
.: {}
f:limits:
.: {}
f:cpu: {}
f:memory: {}
f:requests:
.: {}
f:cpu: {}
f:memory: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:volumeMounts:
.: {}
k:{"mountPath":"/secrets/cloudsql"}:
.: {}
f:mountPath: {}
f:name: {}
f:readOnly: {}
f:dnsPolicy: {}
f:enableServiceLinks: {}
f:nodeSelector:
.: {}
f:non-preemptible: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:terminationGracePeriodSeconds: {}
f:tolerations: {}
f:volumes:
.: {}
k:{"name":"careerassessor-cloudsql-instance-credentials"}:
.: {}
f:name: {}
f:secret:
.: {}
f:defaultMode: {}
f:secretName: {}
manager: kube-controller-manager
operation: Update
time: "2021-08-10T10:59:29Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:conditions:
k:{"type":"PodScheduled"}:
f:message: {}
f:reason: {}
manager: kube-scheduler
operation: Update
time: "2021-08-10T10:59:29Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:message: {}
f:phase: {}
f:reason: {}
f:startTime: {}
manager: kubelet
operation: Update
time: "2021-08-10T11:51:28Z"
name: career-assessor-be-8467d6c885-h27sh
namespace: jyostna1
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: career-assessor-be-8467d6c885
uid: 674b9e8e-420e-44e7-9601-871be01a9fcb
resourceVersion: "48899168"
uid: 8837f88d-7e3e-444f-a804-32a7a6e98c71
spec:
containers:
- args:
- |
xxxx
command:
- /bin/sh
- -c
env:
- name: LOG_TO_CONSOLE
value: "1"
- name: INACTIVITY_PERIOD
- name: USER_EMAIL
value: jyostna@springboard.com
envFrom:
- configMapRef:
name: careerassessor-config
image: us.gcr.io/springboard-production/career_assessor:IP-405-implement-explored-strategy-for-r
imagePullPolicy: Always
name: career-assessor-be
ports:
- containerPort: 8001
name: be-port
protocol: TCP
resources:
limits:
cpu: 3200m
memory: 2400Mi
requests:
cpu: 1600m
memory: 1800Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-xpdwd
readOnly: true
- command:
- /cloud_sql_proxy
- -instances=$(CLOUD_SQL_CONNECTION_NAME)=tcp:0.0.0.0:3306
- -credential_file=/secrets/cloudsql/cloudsql-instance-credentials.json
- -term_timeout=$(CLOUD_SQL_CONNECTION_TIMEOUT)s
env:
- name: CLOUD_SQL_CONNECTION_NAME
valueFrom:
configMapKeyRef:
key: CLOUD_SQL_CONNECTION_NAME
name: careerassessor-config
- name: CLOUD_SQL_CONNECTION_TIMEOUT
valueFrom:
configMapKeyRef:
key: CLOUD_SQL_CONNECTION_TIMEOUT
name: careerassessor-config
image: gcr.io/cloudsql-docker/gce-proxy:1.17
imagePullPolicy: IfNotPresent
name: cloudsql-proxy
resources:
limits:
cpu: 100m
memory: 50Mi
requests:
cpu: 20m
memory: 20Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /secrets/cloudsql
name: careerassessor-cloudsql-instance-credentials
readOnly: true
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-xpdwd
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: gke-cluster-reliable-preemptible-node-4b42c9be-x9qs
nodeSelector:
non-preemptible: "false"
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoSchedule
key: non-preemptible
operator: Equal
value: "false"
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: careerassessor-cloudsql-instance-credentials
secret:
defaultMode: 420
secretName: careerassessor-cloudsql-instance-credentials
- name: default-token-xpdwd
secret:
defaultMode: 420
secretName: default-token-xpdwd
status:
message: Node is shutting, evicting pods
phase: Failed
reason: Shutdown
startTime: "2021-08-10T11:14:30Z"
节点描述
Name: gke-cluster-reliable-preemptible-node-xxxxx
Roles: <none>
Labels: beta.kubernetes.io/arch=xxx
beta.kubernetes.io/instance-type=xxx
beta.kubernetes.io/os=linux
cloud.google.com/gke-boot-disk=pd-standard
cloud.google.com/gke-container-runtime=containerd
cloud.google.com/gke-nodepool=preemptible-nodepool
cloud.google.com/gke-os-distribution=cos
cloud.google.com/gke-preemptible=true
cloud.google.com/machine-family=n1
failure-domain.beta.kubernetes.io/region=us-central1
failure-domain.beta.kubernetes.io/zone=us-central1-a
kubernetes.io/arch=amd64
kubernetes.io/hostname=gke-cluster-reliable-preemptible-node-xxxx
kubernetes.io/os=linux
node.kubernetes.io/instance-type=n1-standard-4
non-preemptible=false
topology.gke.io/zone=us-central1-a
topology.kubernetes.io/region=us-central1
topology.kubernetes.io/zone=us-central1-a
Annotations: container.googleapis.com/instance_id: 7488269578212988511
csi.volume.kubernetes.io/nodeid:
{"pd.csi.storage.gke.io":"projects/playground-206205/zones/us-central1-a/instances/gke-cluster-reliable-preemptible-node-4b42c9be-x9qs"}
node.alpha.kubernetes.io/ttl: 0
node.gke.io/last-applied-node-labels:
cloud.google.com/gke-boot-disk=pd-standard,cloud.google.com/gke-container-runtime=containerd,cloud.google.com/gke-nodepool=preemptible-nod...
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 10 Aug 2021 17:24:03 +0530
Taints: non-preemptible=false:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: gke-cluster-reliable-preemptible-node-4b42c9be-x9qs
AcquireTime: <unset>
RenewTime: Tue, 10 Aug 2021 20:27:03 +0530
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
FrequentDockerRestart False Tue, 10 Aug 2021 20:24:28 +0530 Tue, 10 Aug 2021 17:24:08 +0530 NoFrequentDockerRestart docker is functioning properly
FrequentContainerdRestart False Tue, 10 Aug 2021 20:24:28 +0530 Tue, 10 Aug 2021 17:24:08 +0530 NoFrequentContainerdRestart containerd is functioning properly
FrequentUnregisterNetDevice False Tue, 10 Aug 2021 20:24:28 +0530 Tue, 10 Aug 2021 17:24:08 +0530 NoFrequentUnregisterNetDevice node is functioning properly
CorruptDockerOverlay2 False Tue, 10 Aug 2021 20:24:28 +0530 Tue, 10 Aug 2021 17:24:08 +0530 NoCorruptDockerOverlay2 docker overlay2 is functioning properly
KernelDeadlock False Tue, 10 Aug 2021 20:24:28 +0530 Tue, 10 Aug 2021 17:24:08 +0530 KernelHasNoDeadlock kernel has no deadlock
ReadonlyFilesystem False Tue, 10 Aug 2021 20:24:28 +0530 Tue, 10 Aug 2021 17:24:08 +0530 FilesystemIsNotReadOnly Filesystem is not read-only
FrequentKubeletRestart False Tue, 10 Aug 2021 20:24:28 +0530 Tue, 10 Aug 2021 17:24:08 +0530 NoFrequentKubeletRestart kubelet is functioning properly
NetworkUnavailable False Tue, 10 Aug 2021 17:24:03 +0530 Tue, 10 Aug 2021 17:24:03 +0530 RouteCreated NodeController create implicit route
MemoryPressure False Tue, 10 Aug 2021 20:26:16 +0530 Tue, 10 Aug 2021 17:24:00 +0530 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 10 Aug 2021 20:26:16 +0530 Tue, 10 Aug 2021 17:24:00 +0530 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 10 Aug 2021 20:26:16 +0530 Tue, 10 Aug 2021 17:24:00 +0530 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 10 Aug 2021 20:26:16 +0530 Tue, 10 Aug 2021 17:24:03 +0530 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 10.128.0.100
ExternalIP: 34.133.49.148
InternalDNS: gke-cluster-reliable-preemptible-node-4b42c9be-x9qs.c.playground-206205.internal
Hostname: gke-cluster-reliable-preemptible-node-4b42c9be-x9qs.c.playground-206205.internal
Capacity:
attachable-volumes-gce-pd: 127
cpu: 4
解决方案
感谢您提供所有信息。根据文档并假设您的 GKE 集群是 1.20 版本：
在运行 1.20 或更高版本的抢占式 GKE 节点上，默认启用 kubelet 正常节点关闭功能。结果，kubelet 检测到抢占并优雅地终止 Pod。
对于可抢占节点上的 Pod，不要为 terminateGracePeriodSeconds 指定超过 25 秒，因为这些 Pod 在抢占期间只会收到 25 秒。
使用 taint 和 toleration 的最佳方法是使用在抢占式 VM 上创建的默认标签 - 为抢占式 VM 污染节点：
kubectl taint nodes node-name cloud.google.com/gke-preemptible="true":NoSchedule
为 Pod 添加容忍度：
tolerations:
- key: cloud.google.com/gke-preemptible
operator: Equal
value: "true"
effect: NoSchedule
还：
当 kubelet 在抢占式节点关闭期间终止 Pod 时，它会为 Pod 分配一个失败状态和一个关闭原因。这些 Pod 在下一次垃圾回收期间被清理。您还可以使用以下命令手动删除关闭的 Pod：
kubectl get pods --all-namespaces | grep -i shutdown | awk '{print $1, $2}' | xargs kubectl delete pod -n
请查看解释所有详细信息的完整文档：https ://cloud.google.com/kubernetes-engine/docs/how-to/preemptible-vms
推荐阅读
- postgresql - 执行操作时，Psql shell 没有给我任何确认
- cadence-workflow - 如何在 Cadence 中增加 Persistence QPS 限制
- spring - 在 Tomcat 服务器托管的 Web 应用程序日志中记录 Hibernate 调用的过程的参数值
- python - 如何从子进程读取输出但不会因换行而被阻止
- graphql - 有没有办法在 Microsoft API 中使用 office365 的信息保护？理想情况下是graphql
- kubernetes - 从同一个 YAML 定义文件创建多个 pod
- spring-boot - 使用kotlin在SpringBoot中映射结构接口错误
- google-kubernetes-engine - 使用访问模式在 GKE 上创建 PV 卷的问题
- npm - cljs.closure/build (closure.clj:319 3) 处的执行错误 (ExceptionInfo)。:bundle-cmd :none 失败——带有 webpack 的 clojurescript
- reactjs - 我可以通过 Netlify 使用 react-router-dom 来免费托管我的 react 应用程序吗？