scala - 类型安全配置未在 sbt 程序集中解析
问题描述
我正在使用 typesafe.Config 在我的 Scala 类中加载 application.conf ,并且使用这种方法可以正确解析驻留在 src/main/resources 中的配置文件:
val config = ConfigFactory.load()
config.resolve()
但是,当我尝试使用 spark-submit 运行 jar 文件时,找不到该文件。我使用 sbt 程序集打包了 jar 文件。我该如何解决这种行为?这是我的带有合并策略和依赖项的 build.sbt 文件:
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.9.5"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.5"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.9.5"
dependencyOverrides += "org.scalatest" %% "scalatest" % "3.2.0-SNAP10" % "it, test"
resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven"
resolvers += "confluent" at "http://packages.confluent.io/maven/"
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion,
"org.apache.spark" %% "spark-hive" % sparkVersion,
"org.scalatest" %% "scalatest" % "3.2.0-SNAP10" % "it, test",
"org.scalacheck" %% "scalacheck" % "1.14.0" % "it, test",
"io.kubernetes" % "client-java" % "3.0.0" % "it",
"org.json" % "json" % "20180813",
"io.circe" %% "circe-core" % circeVersion,
"io.circe" %% "circe-generic" % circeVersion,
"io.circe" %% "circe-parser" % circeVersion,
"org.apache.avro" % "avro" % "1.8.2",
"org.apache.spark" %% "spark-avro" % "2.4.0",
"org.apache.logging.log4j" % "log4j-core" % "2.7",
"org.apache.logging.log4j" % "log4j-api" % "2.7",
"io.prestosql" % "presto-jdbc" % "304" % "it, test",
"com.facebook.presto" % "presto-jdbc" % "0.217" % "it, test",
"com.microsoft.azure" % "azure-sqldb-spark" % "1.0.2",
"com.microsoft.sqlserver" % "mssql-jdbc" % "7.2.0.jre8",
"com.typesafe" % "config" % "1.3.3"
)
assemblyJarName in assembly := "spark_mssql_job_avro.jar"
mainClass in assembly := Some("main.SparkJob")
assemblyMergeStrategy in assembly := {
case x if x.endsWith(".conf") => MergeStrategy.discard
case PathList("org", "apache", "spark", "unused", "UnusedStubClass.class") => MergeStrategy.first
case PathList("org", "apache", "commons", "logging", _*) => MergeStrategy.first
case PathList("org", "apache", "commons", "beanutils", _*) => MergeStrategy.first
case PathList("org", "apache", "commons", "collections", _*) => MergeStrategy.first
case PathList("org", "apache", "hadoop", "yarn", _*) => MergeStrategy.first
case PathList("org", "aopalliance", _*) => MergeStrategy.first
case PathList("org", "objenesis", _*) => MergeStrategy.first
case PathList("com", "sun", "jersey", _*) => MergeStrategy.first
case PathList("org", "apache", "hadoop", "yarn", _*) => MergeStrategy.first
case PathList("org", "slf4j", "impl", _*) => MergeStrategy.first
case PathList("com", "codahale", "metrics", _*) => MergeStrategy.first
case PathList("javax", "transaction", _*) => MergeStrategy.first
case PathList("javax", "inject", _*) => MergeStrategy.first
case PathList("javax", "xml", _*) => MergeStrategy.first
case PathList("META-INF", "jersey-module-version") => MergeStrategy.first
case PathList("plugin.xml") => MergeStrategy.first
case PathList("parquet.thrift") => MergeStrategy.first
case PathList("git.properties") => MergeStrategy.first
case PathList("codegen", "config.fmpp") => MergeStrategy.first
case PathList("overview.html") => MergeStrategy.discard
case "application.conf" => MergeStrategy.concat
case x => (assemblyMergeStrategy in assembly).value(x)
}
解决方案
您删除所有.conf
文件(当有多个同名文件时)
case x if x.endsWith(".conf") => MergeStrategy.discard
删除该行应该可以解决问题。以防万一,事后查看 jar 并检查文件是否包含您期望的内容。
推荐阅读
- docker - 使用 Google Cloud 代码在 vscode 中指定本地 docker 容器注册表
- excel - 让所有黄色单元格在范围内的快速方法
- javascript - JS 中的 ReactNative 中未显示图像
- c# - log4net 没有记录任何内容,也没有创建文件
- rabbitmq - Symfony:我如何在 Messenger 中使用远程过程调用(RPC)?
- node.js - node.js 将多个字符串组合成一个多行字符串
- apache-flink - 哪个设置检查点间隔(毫秒)?
- web-scraping - 触发函数应用时 Azure 逻辑应用超时
- java - Edittext 在 android 中接受价格格式
- c# - C#第一次调用https api很慢,但是python没问题