apache-kafka - Druid Kafka 摄取的配置
问题描述
我想设置 kafka druid 摄取,但即使在 common.runtime.properties 中配置并添加 druid-kafka 索引服务后,它仍然给我错误。这个你能帮我吗。我的数据是 csv 格式。
{
"type": "kafka",
"spec": {
"dataSchema": {
"dataSource": "london_crime_by_lsoa",
"parser": {
"type": "string",
"parseSpec": {
"format": "csv",
"dimensionsSpec": {
"dimensions": [
"lsoa_code",
"borough",
"major_category",
"minor_category",
{
"name": "value",
"type": "long"
},
{
"name": "year",
"type": "long"
},
{
"name": "month",
"type": "long"
}
]
},
"timestampSpec": {
"column": "year",
"format": "auto"
},
"columns": [
"lsoa_code",
"borough",
"major_category",
"minor_category",
"value",
"year",
"month"
]
}
},
"metricsSpec": [],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "year",
"queryGranularity": "NONE",
"rollup": false
}
},
"ioConfig": {
"topic": "london_crime_by_lsoa",
"taskDuration": "PT10M",
"useEarliestOffset": "true",
"consumerProperties": {
"bootstrap.servers": "localhost:9092"
}
},
"tuningConfig": {
"type": "kafka",
"maxRowsPerSegment": 500000
}
}
}
执行此命令后:
curl -XPOST -H'Content-Type: application/json' -d @quickstart/tutorial/crime_supervisor.json http://localhost:8090/druid/indexer/v1/supervisor
我收到此错误:
{"error":"Instantiation of [simple type, class org.apache.druid.indexing.kafka.supervisor.KafkaSupervisorSpec] value failed: dataSchema"}
解决方案
我认为您在 JSON 中指定规范的方式存在问题。您必须dataSchema
直接在 JSON 中指定,而不是作为spec
.
这是您应该遵循的格式:
{
"type": "kafka",
"dataSchema": {},
"tuningConfig": {},
"ioConfig": {}
}
推荐阅读
- linkedin - LinkedIn Lead Gen webhook 使用 Google Analytics 参数连接到 Pardot
- bootstrap-4 - 需要帮助配置 Bootstrap
- android - 亚马逊的 DynomoDb 依赖问题
- flutter - 在 Dart / Flutter 中重新实例化 Class 变量
- html - 如何使这个选择元素响应指针?
- snowflake-cloud-data-platform - 无法识别时间戳“xxxx-xx-xxTxx:xx:xx+xxxx”
- python - 如果转置数据集中满足条件,如何聚合?
- python - Django3.2 python迁移未设置默认值
- css - R Shiny CSS:删除特定行之间的空间,同时保持所有其他间距不变
- python - 功率计 - Modbus RS485 - 5 到 6 小时后,设备不再响应