scala - 引起:com.fasterxml.jackson.databind.JsonMappingException:不兼容的杰克逊版本:2.8.9
问题描述
当我df.show()
打印 DataFrame 行的内容时,出现此错误:
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.8.9
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:747)
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)
这就是我创建的方式df
:
object Test extends App {
val spark = SparkSession.builder()
.config("es.nodes", "XXX.XX.XX.XX")
.config("es.port", "9200")
.config("es.nodes.wan.only", "false")
.config("es.resource","myIndex")
.appName("Test")
.master("local[*]")
.getOrCreate()
val df_source = spark
.read.format("org.elasticsearch.spark.sql")
.option("pushdown", "true")
.load("myIndex")
df_source.show(5)
}
我不在我的build.sbt
.
更新:
import sbtassembly.AssemblyPlugin.autoImport.assemblyOption
name := "test"
lazy val spark = "org.apache.spark"
lazy val typesafe = "com.typesafe.akka"
val sparkVersion = "2.2.0"
val elasticSparkVersion = "6.2.4"
val scalaLoggingVersion = "3.7.2"
val slf4jVersion = "1.7.5"
val kafkaVersion = "0.8.0.0"
val akkaVersion = "2.5.9"
val playVersion = "2.6.8"
val sprayVersion = "1.3.2"
val opRabbitVersion = "2.1.0"
val orientdbVersion = "2.2.34"
val livyVersion = "0.5.0-incubating"
val scalaHttpVersion = "2.3.0"
val scoptVersion = "3.3.0"
resolvers ++= Seq(
// repo for op-rabbit client
"SpinGo OSS" at "http://spingo-oss.s3.amazonaws.com/repositories/releases",
"SparkPackagesRepo" at "http://dl.bintray.com/spark-packages/maven",
"cloudera.repo" at "https://repository.cloudera.com/artifactory/cloudera-repos"
)
lazy val commonSettings = Seq(
organization := "org.test",
version := "0.1",
scalaVersion := "2.11.8",
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = true),
assemblyMergeStrategy in assembly := {
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case PathList("reference.conf") => MergeStrategy.concat
case x => MergeStrategy.first
}
)
val sparkSQL = spark %% "spark-sql" % sparkVersion
val sparkGraphx = spark %% "spark-graphx" % sparkVersion
val sparkMLLib = spark %% "spark-mllib" % sparkVersion
val elasticSpark = "org.elasticsearch" % "elasticsearch-hadoop" % elasticSparkVersion
val livyAPI = "org.apache.livy" % "livy-api" % livyVersion
val livyScalaAPI = "org.apache.livy" %% "livy-scala-api" % livyVersion
val livyClientHttp = "org.apache.livy" % "livy-client-http" % livyVersion
val spingoCore = "com.spingo" %% "op-rabbit-core" % opRabbitVersion
val spingoPlayJson = "com.spingo" %% "op-rabbit-play-json" % opRabbitVersion
val spingoJson4s = "com.spingo" %% "op-rabbit-json4s" % opRabbitVersion
val spingoAirbrake = "com.spingo" %% "op-rabbit-airbrake" % opRabbitVersion
val spingoAkkaStream = "com.spingo" %% "op-rabbit-akka-stream" % opRabbitVersion
val orientDB = "com.orientechnologies" % "orientdb-graphdb" % orientdbVersion excludeAll(
ExclusionRule("commons-beanutils", "commons-beanutils-core"),
ExclusionRule("commons-collections", "commons-collections"),
ExclusionRule("commons-logging", "commons-logging"),
ExclusionRule("stax", "stax-api")
)
val scopt = "com.github.scopt" %% "scopt" % scoptVersion
val spray = "io.spray" %% "spray-json" % sprayVersion
val scalaHttp = "org.scalaj" %% "scalaj-http" % scalaHttpVersion
lazy val graph = (project in file("./app"))
.settings(
commonSettings,
libraryDependencies ++= Seq(sparkSQL, sparkGraphx, sparkMLLib, orientDB,
livyAPI, livyScalaAPI, livyClientHttp, scopt,
spingoCore, scalaHttp,
spray, spingoCore, spingoPlayJson, spingoJson4s,
spingoAirbrake, spingoAkkaStream, elasticSpark)
)
dependencyOverrides += "com.typesafe.akka" %% "akka-stream" % akkaVersion
我尝试为 Spark 添加 Jackson 库,但没有解决问题:
val jacksonCore = "com.fasterxml.jackson.core" % "jackson-core" % "2.6.5"
val jacksonDatabind = "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.5"
val jacksonAnnotations = "com.fasterxml.jackson.core" %% "jackson-annotations" % "2.6.5"
val jacksonScala = "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.6.5"
最后,我这样做了(最后两个依赖项由于某种原因无法解决):
dependencyOverrides += "com.typesafe.akka" %% "akka-stream" % akkaVersion
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.9"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.9"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-annotations" % "2.8.9"
dependencyOverrides += "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.8.9"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-paranamer" % "2.8.9"
但现在我得到了错误:
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/scala/DefaultScalaModule$
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.module.scala.DefaultScalaModule$
解决方案
Spark 2.2.0 的 jackson 版本是 2.6.5,看起来您的其他依赖项之一正在使用 jackson 2.8.9,这两个版本不兼容,因此您需要将它们与相同的 jackson 版本对齐。
这个 build.sbt 看起来很成问题,因为你混合了很多可能与 jackson 和其他依赖项不一致的东西。
例如 op-rabbit-json4s 期望 jackson 为 3.5.3,另一方面,我认为 orientdb-graphdb 期望 jackson 的第三个版本(2.2.3)
总之,您需要尽可能多地调整依赖关系,以确保没有冲突。
在这里你可以找到一个有用的插件来检查依赖关系https://github.com/jrudolph/sbt-dependency-graph
推荐阅读
- regex - String.replace 函数返回非字符串输出
- php - 检查数据库的值,同时创建一个 foreach 循环复选框
- metpy - Stationplot barbs 因 AttributeError 而失败:“list”对象没有属性“shape”
- reactjs - 如何将 toMatchSnapshot 与酶一起使用,因此结果类似于带有 react-testing-library 的 container.firstChil
- java - Java 日历没有得到正确的星期几
- java - 为什么在 Effective Java 中推荐使用静态方法?
- python-3.x - 使用 configparser 导入系统路径不起作用
- android - 反应本机linkedin登录正在重定向到url
- javascript - 从数组中删除所有具有特征 X 的元素,除了最新的
- android - Android Spinner 使项目可点击但不可选择