apache-spark - Azure Databricks 增量删除上的广播超时

Question

您好我正在尝试从增量表中删除记录。它不时导致广播超时错误。有人可以帮忙吗

spark.sql(s"""DELETE FROM stg.bl  WHERE concat(key,':',revision) in 
   (Select distinct concat(bl.key,':',bl.revision) from stg.bl bl left semi join
    tgt.bl tgt ON bl.key = tgt.key and bl.revision = tgt.revision)""")

org.apache.spark.SparkException: Could not execute broadcast in 300 secs. 
You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or 
disable broadcast join 
by setting spark.sql.autoBroadcastJoinThreshold to -1

标签： apache-sparkazure-databricksdelta-lake