apache-spark - Spark Streaming 延迟写入接收器

Question

我想在写入流式接收器（即 eventthub）之前引入 1 小时的延迟。如何做到这一点？我尝试了以下方法：

sinkOutput
 .withColumn("Timestamp", current_timestamp())
 .withWatermark("Timestamp", "60 minutes")
 .trigger(Trigger.ProcessingTime("60 minutes"))
 .format("org.apache.spark.sql.eventhubs.EventHubsSourceProvider")
 .option("checkpointLocation", checkpointLocation)
 .options(ehConf.toMap) // EventHubsConf containing the destination EventHub connection string.
 .start()

但它似乎不起作用。有什么我想念的吗？

标签： apache-sparkspark-streamingspark-structured-streaming