databricks - Databricks 连接不能从 intellj 工作？

Question

我正在尝试使用 databricks connect 从 intellj 在 databricks 集群上运行 spark 作业。我遵循以下链接文档。

https://docs.databricks.com/dev-tools/databricks-connect.html

但是我不能让它与 intellj 一起工作，它会抛出异常

21/10/01 18:32:07 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/10/01 18:32:07 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
Exception in thread "main" java.lang.NoSuchFieldError: JAVA_9
    at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:207)
    at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
    at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:95)
    at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:443)
    at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:384)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:432)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:262)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:291)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:495)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2834)

我找不到解决方法，因为文档没有明确说明我从 intellj 交叉检查了它指向由 (databricks-connect get-jar-dir) 返回的正确 jar 目录。任何线索都会有帮助吗？

注意：databricks-connect 测试返回成功

标签： databricksintellij-14databricks-connectaws-databricks