elasticsearch - 如何在 ECK 上为 kafka 输入配置 Filebeat?
问题描述
我在 Kubernetes 上运行 Elasticsearch 和 Kibana。两者均由 ECK 创建。现在我尝试将 Filebeat 添加到其中并将其配置为索引来自 Kafka 主题的数据。这是我目前的配置:
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: my-filebeat
namespace: my-namespace
spec:
type: filebeat
version: 7.10.2
elasticsearchRef:
name: my-elastic
kibanaRef:
name: my-kibana
config:
filebeat.inputs:
- type: kafka
hosts:
- host1:9092
- host2:9092
- host3:9092
topics: ["my.topic"]
group_id: "my_group_id"
index: "my_index"
deployment:
podTemplate:
spec:
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
securityContext:
runAsUser: 0
containers:
- name: filebeat
在 pod 的日志中,我可以看到如下条目
log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2470,"time":{"ms":192}},"total":{"ticks":7760,"time":{"ms":367},"value":7760},"user":{"ticks":5290,"time":{"ms":175}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":13},"info":{"ephemeral_id":"5ce8521c-f237-4994-a02e-dd11dfd31b09","uptime":{"ms":181997}},"memstats":{"gc_next":23678528,"memory_alloc":15320760,"memory_total":459895768},"runtime":{"goroutines":106}},"filebeat":{"harvester":{"open_files":0,"running":0},"inputs":{"kafka":{"bytes_read":46510,"bytes_write":37226}}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":1.18,"15":0.77,"5":0.97,"norm":{"1":0.0738,"15":0.0481,"5":0.0606}}}}}}
也不存在错误条目。所以我假设与 Kafka 的连接有效。不幸的是,上面指定的 my_index 中没有数据。我做错了什么?
解决方案
我猜你无法连接到输出中提到的 Elasticsearch。
根据文档,ECK 保护部署的 Elasticsearch 并将其存储在 Kubernetes Secrets 中。
https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-beat-configuration.html
推荐阅读
- ios - 如何监听 ObservableObject
- vue.js - 如何测试从模板将参数传递给 getter 的组件?
- reactjs - React Redux 在函数/生命周期钩子中等待 ui 更新
- excel - AddPicture 需要为文件路径引用单元格并调整图像大小
- amazon-web-services - 如何使用 aws sdk v2 for java 扫描 dynamodb?
- azure-data-lake-gen2 - Delta Lake 与 Azure Data Lake Store Gen2
- node.js - npm run watch 在 Laravel 6.0 中不起作用
- node.js - express-validator 可选字段
- sql - 从一张表创建左右数据集
- ruby-on-rails - ActiveRecord::RecordInvalid:验证失败