web-dev-qa-db-ja.com

Spark2.1.0互換性のないJacksonバージョン2.7.6

私は簡単なspark intellijの例を実行しようとしていますが、そのようなエラーが発生します：

Exception in thread "main" Java.lang.ExceptionInInitializerError
at org.Apache.spark.SparkContext.withScope(SparkContext.scala:701)
at org.Apache.spark.SparkContext.textFile(SparkContext.scala:819)
at spark.test$.main(test.scala:19)
at spark.test.main(test.scala)
at Sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at Sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.Java:62)
at Sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.Java:43)
at Java.lang.reflect.Method.invoke(Method.Java:498)
at com.intellij.rt.execution.application.AppMain.main(AppMain.Java:147)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.7.6
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.Java:730)
at org.Apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82)
at org.Apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)

私はジャクソンの依存関係を更新しようとしましたが、うまくいかないようです、私はこれをします：

libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7"
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"

それでも同じエラーメッセージが表示されますが、誰かがエラーを修正するのに役立ちますか？

spark=サンプルコード：

object test {
def main(args: Array[String]): Unit = {
    if (args.length < 1) {
        System.err.println("Usage: <file>")
        System.exit(1)
    }

    val conf = new SparkConf()
    val sc = new SparkContext("local","wordcount",conf)
    val line = sc.textFile(args(0))

    line.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_).collect().foreach(println)

    sc.stop()
    }
}

そして、ここに私のbuilt.sbtがあります：

name := "testSpark2"

version := "1.0"

scalaVersion := "2.11.8"


libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7"
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"

libraryDependencies += "org.Apache.spark" % "spark-core_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-mllib_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-repl_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-streaming-flume_2.10" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-sql_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-network-shuffle_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-Hive_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-streaming-flume-Assembly_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-mesos_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-graphx_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-catalyst_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-launcher_2.11" % "2.1.0"
scalaapache-sparkjacksonsbtincompatibletypeerror
32
Yang

Spark 2.1.0には、推移的な依存関係としてcom.fasterxml.jackson.coreが含まれています。そのため、libraryDependenciesに含める必要はありません。

ただし、異なるcom.fasterxml.jackson.core依存関係のバージョンを追加する場合は、それらをオーバーライドする必要があります。このような：

name := "testSpark2"

version := "1.0"

scalaVersion := "2.11.8"


dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7"

libraryDependencies += "org.Apache.spark" % "spark-core_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-mllib_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-repl_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-streaming-flume_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-sql_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-network-shuffle_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-Hive_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-streaming-flume-Assembly_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-mesos_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-graphx_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-catalyst_2.11" % "2.1.0"
libraryDependencies += "org.Apache.spark" % "spark-launcher_2.11" % "2.1.0"

したがって、build.sbtを上記のように変更すると、期待どおりに動作します。

私はそれが役立つことを願っています！

45
himanshuIIITian

ご参考までに。私の場合、spark and kafka-streamsアプリでは、kafka-streamsはcom.fasterxml.jackson.core 2.8.5を使用します。以下のexcludeを追加すると、問題が修正されました

（グラドル）

compile (group: "org.Apache.kafka", name: "kafka-streams", version: "0.11.0.0"){
    exclude group:"com.fasterxml.jackson.core"
}
7
Leon

ResolutionStrategyを使用したGradleによるソリューション（ https://docs.gradle.org/current/dsl/org.gradle.api.artifacts.ResolutionStrategy.html ）：

configurations {

    all {

        resolutionStrategy {
            force 'com.fasterxml.jackson.core:jackson-core:2.4.4', 'com.fasterxml.jackson.core:jackson-databind:2.4.4', 'com.fasterxml.jackson.core:jackson-annotations:2.4.4'
        }
    }
}
1
Thomas Decaux