时间戳

首页 > 解决方案 > 如何在kafka中禁用标头消费

apache-kafka - 如何在kafka中禁用标头消费

问题描述

是否有任何选项可以禁用消费者使用的 Kafka 标头。在我的例子中，我编写了一个消费者来消费来自上游系统发布的 Kafka 主题的消息。我的处理不需要来自标头的任何信息，并且已发布的标头很重（大于消息本身的大小）。所以我的消费者花费的时间比预期的要长。

我只能使用留下标头的消息内容的任何选项，这样可以节省通过网络传输标头并在消费者处反序列化它们的时间。感谢您的帮助。

标签： apache-kafkakafka-consumer-api

解决方案

Every message is a Record with Headers (as of Kafka 0.11).

length: varint
attributes: int8
    bit 0~7: unused
timestampDelta: varint
offsetDelta: varint
keyLength: varint
key: byte[]
valueLen: varint
value: byte[]
Headers => [Header]

Record Header

headerKeyLength: varint
headerKey: String
headerValueLength: varint
Value: byte[]

Even if you ignore deserializing them, they will still be sent over the wire as part of the Record's TCP packet body.

You could try using a Kafka 0.10.2 client version, for example, which might drop the header entirely, because they just weren't part of the API, but YMMV.

As mentioned in the comments, the most reliable way here would be to stop sending such heavy information in the upstream application. Or the middle-ground would be to compress, and/or binary encode that data.

推荐阅读