spring-cloud - 无法解码密钥的 json 类型:Spring Cloud Data Flow 流中的 file_name
问题描述
我使用 Spring Cloud Data Flow 设置读取 CSV 文件的流,使用自定义处理器对其进行转换并记录它:
stream create --name testsourcecsv --definition "file --mode=lines --directory=D:/toto/ --file.filename-pattern=adresses-28.csv --maxMessages=1000 | csvToMap --spring.cloud.stream.bindings.output.content-type=application/json | log --spring.cloud.stream.bindings.input.content-type=application/json" --deploy
文件和 csvToMap 应用程序工作正常,但在日志应用程序中我看到这种异常,对于每条记录:
2019-12-03 11:32:46.500 ERROR 1328 --- [container-0-C-1] o.s.c.s.b.k.KafkaMessageChannelBinder$5 : Could not decode json type: adresses-28.csv for key: file_name
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'adresses': was expecting ('true', 'false' or 'null')
at [Source: (byte[])"adresses-28.csv"; line: 1, column: 10]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) ~[jackson-core-2.9.9.jar!/:2.9.9]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:703) ~[jackson-core-2.9.9.jar!/:2.9.9]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3532) ~[jackson-core-2.9.9.jar!/:2.9.9]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2627) ~[jackson-core-2.9.9.jar!/:2.9.9]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:832) ~[jackson-core-2.9.9.jar!/:2.9.9]
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:729) ~[jackson-core-2.9.9.jar!/:2.9.9]
at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4141) ~[jackson-databind-2.9.9.jar!/:2.9.9]
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4000) ~[jackson-databind-2.9.9.jar!/:2.9.9]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3091) ~[jackson-databind-2.9.9.jar!/:2.9.9]
at org.springframework.cloud.stream.binder.kafka.BinderHeaderMapper.lambda$toHeaders$1(BinderHeaderMapper.java:268) ~[spring-cloud-stream-binder-kafka-2.1.4.RELEASE.jar!/:2.1.4.RELEASE]
at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_202]
at org.springframework.cloud.stream.binder.kafka.BinderHeaderMapper.toHeaders(BinderHeaderMapper.java:251) ~[spring-cloud-stream-binder-kafka-2.1.4.RELEASE.jar!/:2.1.4.RELEASE]
对于 file_relativePath 标头也会引发此异常。我不明白为什么 spring-kafka 试图将它们读取为 JSON。
此外,日志接收器以正确的方式记录我的记录:
2019-12-03 11:32:46.516 INFO 1328 --- [container-0-C-1] log-sink : {"code_postal":"28200","id_fantoir":"28211_0127","source_nom_voie":"inconnue","numero":"1","code_insee":28211,"lon":1.260462,"code_insee_ancienne_commune":"","nom_afnor":"RUE DU VIEUX MOULIN","nom_voie":"Rue du Vieux Moulin","nom_ld":"","libelle_acheminement":"LOGRON","source_position":"inconnue","nom_commune":"Logron","nom_ancienne_commune":"","x":570633.27,"y":6784246.2,"alias":"","id":"28211_0127_00001","rep":"","lat":48.145756}
我在我的 csvToMap 处理器中记录了 kafka 标头以进行调试,给了我:
2019-12-03 11:32:37.042 INFO 10788 --- [container-0-C-1] c.d.streams.processor.CsvToMapProcessor : headers {sequenceNumber=152963, file_name=adresses-28.csv, sequenceSize=0, deliveryAttempt=1, kafka_timestampType=CREATE_TIME, file_originalFile=NonTrustedHeaderType [headerValue="D:\\toto\\adresses-28.csv", untrustedType=java.io.File], kafka_receivedMessageKey=null, kafka_receivedTopic=testsourcecsv.file, file_relativePath=adresses-28.csv, kafka_offset=430949, scst_nativeHeadersPresent=true, kafka_consumer=org.apache.kafka.clients.consumer.KafkaConsumer@7c3e63db, correlationId=9547c02d-e617-d981-f9b5-8df231530f66, kafka_receivedPartitionId=0, contentType=text/plain, kafka_receivedTimestamp=1575299282558, kafka_groupId=testsourcecsv}
所以我绝对不明白为什么日志接收器会尝试解码 file_name 和 file_relativePath 标头。
我设置了一个本地环境:
- Windows 7的
- Spring CDF 服务器 v 2.2.1.REALEASE
- Spring Cloud Skipper v 2.1.2.RELEASE
- Spring CDF 外壳 v 2.2.1.RELEASE
- 卡夫卡 2.12-2.3.0
我的 csvToMap 处理器定义如下:
@Component
public class CsvToMapProcessor {
private static final Logger LOGGER = LoggerFactory.getLogger(CsvToMapProcessor.class);
@Autowired
@Qualifier("csvMapper")
private ObjectReader csvMapper;
@Autowired
@Qualifier("jsonWriter")
private ObjectWriter jsonWriter;
@Transformer(inputChannel = Processor.INPUT, outputChannel = Processor.OUTPUT)
public Map<String, Object> transform(String csvLine, @Headers Map<String, Object> headers) {
try {
LOGGER.info("headers {}", headers);
Map<String, Object> map = csvMapper.readValue(csvLine);
return map;
} catch (JsonProcessingException e) {
LOGGER.error("An error occurs while reading CSV line {} : {}", csvLine, e.getMessage());
LOGGER.debug(e.getMessage(), e);
return null;
}
}
}
与此父母:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.2.1.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
而这个 Spring 云版本:
<spring-cloud.version>Hoxton.RELEASE</spring-cloud.version>
我做错了什么导致这个问题?
解决方案
所以 :
- 与 spring-cloud Hoxton 版本一起使用,spring-cloud-stream 版本为 3.0.0.RELEASE :
[INFO] +- org.springframework.cloud:spring-cloud-starter-stream-kafka:jar:3.0.0.RELEASE:compile
[INFO] | \- org.springframework.cloud:spring-cloud-stream-binder-kafka:jar:3.0.0.RELEASE:compile
[INFO] | +- org.springframework.cloud:spring-cloud-stream-binder-kafka-core:jar:3.0.0.RELEASE:compile
[INFO] | | \- org.springframework.integration:spring-integration-kafka:jar:3.2.1.RELEASE:compile
[INFO] | \- org.springframework.kafka:spring-kafka:jar:2.3.3.RELEASE:compile
- log-sink-app 2.1.2 使用 spring-cloud-stream v 2.1.4.RELEASE :
[INFO] +- org.springframework.cloud:spring-cloud-starter-stream-kafka:jar:2.1.4.RELEASE:compile
[INFO] | \- org.springframework.cloud:spring-cloud-stream-binder-kafka:jar:2.1.4.RELEASE:compile
[INFO] | +- org.springframework.cloud:spring-cloud-stream-binder-kafka-core:jar:2.1.4.RELEASE:compile
[INFO] | | \- org.springframework.integration:spring-integration-kafka:jar:3.1.0.RELEASE:compile
[INFO] | \- org.springframework.kafka:spring-kafka:jar:2.2.8.RELEASE:compile
正如spring-kafka 2.3.3 文档DefaultKafkaHeaderMapper.setEncodeStrings 方法所说:
如果出站记录的使用者使用 Spring for Apache Kafka 版本低于 2.3,则设置为 true
log-sink 应用程序实际上使用 spring-kafka v 2.2.8,所以我必须使用自定义的标头映射器将其设置为 true:
@Bean("kafkaBinderHeaderMapper")
public KafkaHeaderMapper kafkaBinderHeaderMapper() {
DefaultKafkaHeaderMapper mapper = new DefaultKafkaHeaderMapper();
mapper.setEncodeStrings(true);
return mapper;
}
但如果我这样做,日志接收器不会记录任何内容,因为它无法理解由 DefaultKafkaHeaderMapper 编码的 contentType 标头。并且团队提供了一个BinderHeaderMapper来解决这个问题:
Apache Kafka 的自定义标头映射器。这与 spring Kafka 中的 DefaultKafkaHeaderMapper 相同。这是为了解决 Spring Cloud Stream 3.0.x 和 2.x 应用程序之间的一些互操作性问题,其中在标头中作为常规 MimeType 传递的 mime 类型未正确反序列化
所以我必须在我的应用程序中配置一个自定义 BinderHeaderMapper :
@Bean("kafkaBinderHeaderMapper")
public KafkaHeaderMapper kafkaBinderHeaderMapper() {
BinderHeaderMapper mapper = new BinderHeaderMapper();
mapper.setEncodeStrings(true);
return mapper;
}
一切正常。
推荐阅读
- arduino-ide - 如何在 WiFi softAP() 中设置 DHCP IP 地址租用范围?
- elixir - Ecto 寻找 field_id 而不是 field
- rabbitmq - Publisher 在 Rabbitmq 中确认的确切内容是什么?
- php - 重定向页面无法正常工作 - 获取数据并重定向
- python-3.x - 使用深度学习的小目标检测
- java - Keycloak:验证用 JAVA 编写的 Rest API
- webkitspeechrecognition - 如何在不请求麦克风许可的情况下重新启动 webkitSpeechRecognition?
- android - 如何使用新的 `dagger-android` 将活动传递给 dagger 模块
- arrays - 将数组存储到我在 bundle 中创建的 answer.json 文件
- ms-access - 连接富文本字段 (HTML) 并在 Access 表单上显示结果