时间戳

首页 > 解决方案 > kafka jdbc sink连接器：创建或更新一个连接器时重新创建所有其他连接器

apache-kafka - kafka jdbc sink连接器：创建或更新一个连接器时重新创建所有其他连接器

问题描述

我有大约 500 个连接器，每次创建新连接器时，我都必须等待很长时间才能重新创建以前的连接器。它太慢了。

sink.properties 如下

{
    "name": "saas-saas_order-worder_fee_price-sink",
    "config": {
        "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
        "errors.log.include.messages": "true",
        "tasks.max": "1",
        "topics": "saasosvc-worder_fee_price",
        "transforms": "unwrap",
        "auto.evolve": "true",
        "name": "saas-saas_order-worder_fee_price-sink",
        "transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope",
        "auto.create": "true",
        "connection.url": "jdbc:postgresql://postgres:5432/saas?user=postgresuser&password=postgrespw",
        "errors.log.enable": "true",
        "insert.mode": "upsert",
        "pk.mode": "record_value",
        "pk.fields": "id"
    }
}

一些日志如下

> 2019-06-12 05:44:42,242 INFO   || 
> WorkerSourceTask{id=saas-com_dyrs_mtsp_furnituresaleservice-source-0}
> flushing 0 outstanding messages for offset commit  
> [org.apache.kafka.connect.runtime.WorkerSourceTask]
>     2019-06-12 05:44:45,950 INFO   ||  WorkerSinkTask{id=saas-appconstruction-appconstruction_requestrecord-sink-0}
> Committing offsets asynchronously using sequence number 20376:
> {appssvc-appconstruction_requestrecord-0=OffsetAndMetadata{offset=2340,
> leaderEpoch=null, metadata=''}}  
> [org.apache.kafka.connect.runtime.WorkerSinkTask]
>     2019-06-12 05:45:18,242 INFO   ||  Connector saas-saas_order-worder_fee_price-sink config updated  
> [org.apache.kafka.connect.runtime.distributed.DistributedHerder]
>     2019-06-12 05:45:18,745 INFO   ||  Rebalance started   [org.apache.kafka.connect.runtime.distributed.DistributedHerder]
>     2019-06-12 05:45:18,747 INFO   ||  Stopping connector saas-order_base-product_vs_install-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,747 INFO   ||  Stopping connector saas-dyrs_complainservice-complain_type_record-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-appcustomerself-appcustomerself_appversionuser-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-com_dyrs_mtsp_changeservice-replace_product-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-com_dyrs_mtsp_designservice-version_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-dyrs_settlementservice-source  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-appcustomerself-appcustomerself_advice-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,748 INFO   ||  Stopping connector saas-im-im_accesstoken-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,761 INFO   ||  172.31.206.219 - - [12/Jun/2019:05:45:18 +0000] "POST /connectors/ HTTP/1.1" 201 604  549
> [org.apache.kafka.connect.runtime.rest.RestServer]
>     2019-06-12 05:45:18,775 INFO   ||  Stopped connector saas-im-im_accesstoken-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,776 INFO   ||  Stopping connector saas-saas_order-dd_order-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,778 INFO   ||  Stopped connector saas-dyrs_settlementservice-source  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,780 INFO   ||  Stopping connector saas-dyrs_settlementservice-balance_info_detail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,781 INFO   ||  Stopped connector saas-com_dyrs_mtsp_designservice-version_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,781 INFO   ||  Stopping connector saas-dyrs_authorityservice-pro_city_area-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,781 INFO   ||  Stopped connector saas-dyrs_complainservice-complain_type_record-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,781 INFO   ||  Stopping connector saas-constructionconfig-constructionconfig_checkmanagedetailstandard-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,799 INFO   ||  Stopped connector saas-saas_order-dd_order-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,799 INFO   ||  Stopping connector saas-saas_order-goods-sink   [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,857 INFO   ||  Stopped connector saas-saas_order-goods-sink   [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,857 INFO   ||  Stopping connector saas-com_dyrs_mtsp_businessopportunityservice-business_process_operator-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,860 INFO   ||  Stopped connector saas-appcustomerself-appcustomerself_appversionuser-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,860 INFO   ||  Stopping connector saas-com_dyrs_mtsp_changeservice-custom_wood_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,862 INFO   ||  Stopped connector saas-order_base-product_vs_install-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,863 INFO   ||  Stopping connector saas-com_dyrs_mtsp_quotelistservice-pre_quote_detail_tab_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,865 INFO   ||  Stopped connector saas-com_dyrs_mtsp_changeservice-replace_product-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,868 INFO   ||  Stopping connector saas-com_dyrs_mtsp_quoteservice-program_template_tab-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,870 INFO   ||  Stopped connector saas-dyrs_settlementservice-balance_info_detail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,870 INFO   ||  Stopping connector saas-dyrs_settlementservice-balance_config-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,866 INFO   ||  Stopped connector saas-appcustomerself-appcustomerself_advice-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,873 INFO   ||  Stopping connector saas-dyrs_authorityservice-account_login_fail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,872 INFO   ||  Stopped connector saas-dyrs_authorityservice-pro_city_area-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,873 INFO   ||  Stopping connector saas-com_dyrs_mtsp_dataexpansionservice-dynamic_tabpk-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,875 INFO   ||  Stopped connector saas-constructionconfig-constructionconfig_checkmanagedetailstandard-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,875 INFO   ||  Stopping connector saas-com_dyrs_mtsp_promotionservice-offer_content-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,884 INFO   ||  Stopped connector saas-com_dyrs_mtsp_businessopportunityservice-business_process_operator-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,885 INFO   ||  Stopping connector saas-com_dyrs_mtsp_businessopportunityservice-personal_clue_limit-sink
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,891 INFO   ||  Stopped connector saas-com_dyrs_mtsp_changeservice-custom_wood_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,891 INFO   ||  Stopping connector saas-dyrs_settlementservice-balance_operation-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,902 INFO   ||  Stopped connector saas-com_dyrs_mtsp_quotelistservice-pre_quote_detail_tab_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,902 INFO   ||  Stopping connector saas-order_base-unit_vs_unit-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,904 INFO   ||  Stopped connector saas-com_dyrs_mtsp_dataexpansionservice-dynamic_tabpk-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,905 INFO   ||  Stopped connector saas-com_dyrs_mtsp_quoteservice-program_template_tab-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,907 INFO   ||  Stopping connector saas-com_dyrs_mtsp_fieldassociationservice-source  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,904 INFO   ||  Stopped connector saas-dyrs_authorityservice-account_login_fail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,907 INFO   ||  Stopping connector saas-customerself-customerself_msgsendrecorddetail-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,908 INFO   ||  Stopping connector saas-finereport-constructionprocess_constructioninfoprocesscheck-sink 
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:18,913 INFO   ||  Stopped connector saas-dyrs_settlementservice-balance_config-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,134 INFO   ||  Stopped connector saas-order_base-unit_vs_unit-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,136 INFO   ||  Stopping connector saas-com_dyrs_mtsp_furnituresaleservice-fur_sale_contract_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,139 INFO   ||  Stopping connector saas-com_dyrs_mtsp_businessopportunityservice-call_record-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,154 INFO   ||  Stopped connector saas-com_dyrs_mtsp_furnituresaleservice-fur_sale_contract_info-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,154 INFO   ||  Stopping connector saas-com_dyrs_mtsp_fieldassociationservice-pre_filed-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,166 INFO   ||  Stopped connector saas-com_dyrs_mtsp_promotionservice-offer_content-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,166 INFO   ||  Stopping connector saas-runmonitoring-runmonitoring_browsinghistory-sink  
> [org.apache.kafka.connect.runtime.Worker]
>     2019-06-12 05:45:19,167 INFO   ||  Stopped connector saas-com_dyrs_mtsp_fieldassociationservice-pre_filed-sink  
> [org.apache.kafka.connect.runtime.Worker]

谢谢

标签： apache-kafkaapache-kafka-connectconfluent-platform

解决方案

“停止世界”再平衡是 Kafka Connect 的一个已知问题。好消息是 Apache Kafka 2.3 中的KIP-415有一个新的增量重新平衡功能，它应该会让事情变得更好。

与此同时，唯一的其他选择是对您的 Kafka Connect 工作人员进行分区并拥有单独的集群，将 500 个连接器拆分到它们之上（例如，按功能、类型或其他任意因素）。

推荐阅读