design-patterns - Consumer Producer design

Question

I am looking for advice on how the following problem is handled in solutions that use Pulsar/kafka.

The scenario is: A producer is sending messages (in JSON) and the consumer is taking them and inserting data into database tables (specified in the message).

Suddenly the producer changes the structure of the data sent in the messages (let's say because the table structure in the database has a new column).

So for a moment, the queue has messages with the old data structure and now starts receiving messages with new data structure.

My doubt is regarding how the consumer should handle this scenario. What to do with the messages with old structure that are now invalid since they cannot be inserted in the database table since the table structure changed. Retry and then permanently fail (dead letter Q?).

Also, do you usually opt to sent the metadata along with your messages or do you normally handle this in a separate topic or other form.

Thanks for any advice

标签： design-patternsapache-kafkamessage-queueproducer-consumerapache-pulsar

Answer 1

所描述的问题主要与外部系统有关，这将需要在生产者端进行某种看门/预验证，以了解如何使用数据来防止这种情况发生。不幸的是，这引入了紧密耦合，因此如果没有它，您必须显式编写消费者代码以进行强大的消息转换和异常处理，可能包括一种版本号或显式模式，如 Confluent Schema Registry 提供的每条消息（也可能是 Pulsar 的 Schema Registry 功能）