kubernetes - Spring-Kafka消费者协调器不可用或持续循环无效
问题描述
我在 Kubernetes 节点中有三个 pod。
Pod-1:单节点 Kafka 代理
Pod-2:单节点 Zookeeper
Pod-3:多个启用 Spring Boot Kafka 的应用程序
早些时候,我们在 Docker 上使用了类似的设置。在我们转移到 Kubernetes 设置后,我们在 Pod-3 中的所有应用程序都开始出现这种错误
[2020-01-23 06:37:05.267] :: [INFO ] :: o.a.k.c.c.i.AbstractCoordinator.markCoordinatorUnknown.729 :: [Consumer clientId=consumer-2, groupId=OperationResponseConsumer] Group coordinator 9.x.x.x:31865 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
[2020-01-23 06:37:05.386] :: [INFO ] :: o.a.k.c.c.i.AbstractCoordinator.onSuccess.677 :: [Consumer clientId=consumer-2, groupId=OperationResponseConsumer] Discovered group coordinator 9.x.x.x:31865 (id: 2147483646 rack: null)
这一直在循环,没有一个消费者会关注主题。充其量,消费者群体的依恋是断断续续的。这可能是什么?
卡夫卡:2.2 弹簧启动:2.1.5
- - 编辑
在经纪人方面,我不断收到这些日志
[2020-01-23 07:34:15,083] INFO [GroupCoordinator 1]: Member consumer-2-d2102198-df08-4009-9ba2-85350a2e9240 in group openapi-handler-consumer has failed, removing it from the group (kafka.coordinator.group.GroupCoordinator)
[2020-01-23 07:34:15,083] INFO [GroupCoordinator 1]: Preparing to rebalance group openapi-handler-consumer in state PreparingRebalance with old generation 3 (__consumer_offsets-24) (reason: removing member consumer-2-d2102198-df08-4009-9ba2-85350a2e9240 on heartbeat expiration) (kafka.coordinator.group.GroupCoordinator)
- - - - - - - 编辑
这是我们用于 Kafka 的部署描述符
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: 2020-01-23T09:04:18Z
generation: 1
labels:
app: kafka
name: xip-generator-kafka
namespace: xip-dev-env
resourceVersion: "53066450"
selfLink: /apis/extensions/v1beta1/namespaces/xip-dev-env/deployments/xip-generator-kafka
uid: 5664311b-3dbf-11ea-b0f2-005056ae16bc
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: kafka
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: kafka
spec:
containers:
- env:
- name: KAFKA_ADVERTISED_HOST_NAME
value: dip-generator-kafka-svc
- name: KAFKA_ADVERTISED_PORT
value: "9092"
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_ZOOKEEPER_CONNECT
value: xip-generator-zookeeper-svc:2181
- name: KAFKA_NUM_PARTITIONS
value: "3"
image: wurstmeister/kafka
imagePullPolicy: IfNotPresent
name: kafka
ports:
- containerPort: 9092
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2020-01-23T09:04:18Z
lastUpdateTime: 2020-01-23T09:04:21Z
message: ReplicaSet "xip-generator-kafka-76cf94fcdb" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: 2020-01-27T07:24:36Z
lastUpdateTime: 2020-01-27T07:24:36Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
卡夫卡服务 yml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: 2020-01-23T09:04:18Z
labels:
app: kafka
name: xip-generator-kafka-svc
namespace: xip-dev-env
resourceVersion: "52499070"
selfLink: /api/v1/namespaces/xip-dev-env/services/xip-generator-kafka-svc
uid: 56575969-3dbf-11ea-b0f2-005056ae16bc
spec:
clusterIP: 10.x.x.x
externalTrafficPolicy: Cluster
ports:
- name: kafka-port
nodePort: 31865
port: 9092
protocol: TCP
targetPort: 9092
selector:
app: kafka
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
消费者应用部署:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
kubectl.kubernetes.io/last-applied-configuration:
creationTimestamp: 2020-01-27T06:13:38Z
generation: 1
labels:
app: add-devops-template
name: xip-generator-add-devops-template
namespace: xip-dev-env
resourceVersion: "53059193"
selfLink: /apis/extensions/v1beta1/namespaces/xip-dev-env/deployments/xip-generator-add-devops-template
uid: 287d2c61-40cc-11ea-b0f2-005056ae16bc
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: add-devops-template
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: add-devops-template
spec:
containers:
- env:
- name: kafka.serverURL
value: xip-generator-kafka-svc:9092
- name: dmb.generator.mount.path
value: /opt/code-engine/generated-code
- name: spring.kafka.bootstrap-servers
value: xip-generator-kafka-svc:9092
- name: spring.data.mongodb.host
value: mongourl
- name: spring.data.mongodb.port
value: "27018"
- name: spring.data.mongodb.username
value: root
- name: spring.data.mongodb.password
value: xxxxxxxxxxxxxx
- name: spring.data.mongodb.database
value: code_engine_dev
- name: spring.data.mongodb.authentication-database
value: admin
- name: LOGSTASH
value: logurl
- name: vaultURL
value: vaulturl
- name: vaultSecret
value: adminvault
- name: vaultSecretFileName
value: namefile
- name: templateName
value: add-devops.template
- name: gitRepoToken
value: token
- name: gitRepoUserName
value: username
- name: templateDeploymentDetailsRepoBranch
value: development
- name: templateDeploymentDetailsRepoUrl
value: url.git
image: imagename
imagePullPolicy: IfNotPresent
name: add-devops-template
ports:
- containerPort: 8080
protocol: TCP
resources: {}
securityContext:
runAsUser: 0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/code-engine/generated-code
name: xip-dev-pv-generatedcode
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: nexusegistrykey
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: xip-dev-pv-generatedcode
persistentVolumeClaim:
claimName: xip-dev-generatedcode-pvc
- name: xip-dev-pv-logs
persistentVolumeClaim:
claimName: xip-dev-logs-pvc
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2020-01-27T06:13:57Z
lastUpdateTime: 2020-01-27T06:13:57Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: 2020-01-27T06:13:38Z
lastUpdateTime: 2020-01-27T06:13:57Z
message: ReplicaSet "xip-generator-add-devops-template-7df595864c" has successfully
progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
解决方案
推荐阅读
- python - 在 Python 中使用 RarFile 提取单个文件时出现 BadRarFile
- python - Raspberry Pi 4 控制伺服而不晃动
- go - ERR 未知命令`publish` Miniredis Golang
- elixir - Elixir 更改目录并运行 shell 脚本
- r - 如何在计算 R 中的另一列时计算总数和百分比?
- python - 基于 Python 中 fetchall 结果的 MySQL 更新或插入
- c# - C#中抛出异常后如何继续执行代码?
- django - Django - IIS 部署 - 找不到 scriptProcessor
- python-3.x - Spacy 3.0 训练自定义 NER --> 验证此自定义 NER 模型
- java - java.lang.ClassCastException:类 java.lang.String 无法转换为 MidTermProject 类