web-dev-qa-db-ja.com

spark + sbt-アセンブリ： "重複排除：次のファイルの内容が異なります"

sparkアプリケーションを実行し、テストクラスをファットジャーにパックしたい。奇妙なことに、「sbt Assembly」を正常に実行しましたが、「sbt test：Assembly」を実行すると失敗しました。

sbt-Assembly：テストクラスを含む を試しましたが、私の場合は機能しませんでした。

SBTバージョン：0.13.8

build.sbt：

import sbtassembly.AssemblyPlugin._

name := "Assembly-test"

version := "1.0"

scalaVersion := "2.10.5"

libraryDependencies ++= Seq(
  ("org.Apache.spark" % "spark-core_2.10" % "1.3.1" % Provided)
    .exclude("org.mortbay.jetty", "servlet-api").
    exclude("commons-beanutils", "commons-beanutils-core").
    exclude("commons-collections", "commons-collections").
    exclude("commons-logging", "commons-logging").
    exclude("com.esotericsoftware.minlog", "minlog").exclude("com.codahale.metrics", "metrics-core"),
  "org.json4s" % "json4s-jackson_2.10" % "3.2.10" % Provided,
  "com.google.inject" % "guice" % "4.0"
)

Project.inConfig(Test)(assemblySettings)
apache-sparksbtsbt-assembly
10
Grant

以下のsparkアプリで行ったように、アセンブリでmergeStrateyを定義する必要があります。

mergeStrategy in Assembly <<= (mergeStrategy in Assembly) { (old) =>
  {
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
    case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
    case PathList("org", "Apache", xs @ _*) => MergeStrategy.last
    case PathList("com", "google", xs @ _*) => MergeStrategy.last
    case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
    case PathList("com", "codahale", xs @ _*) => MergeStrategy.last
    case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
    case "about.html" => MergeStrategy.rename
    case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last
    case "META-INF/mailcap" => MergeStrategy.last
    case "META-INF/mimetypes.default" => MergeStrategy.last
    case "plugin.properties" => MergeStrategy.last
    case "log4j.properties" => MergeStrategy.last
    case x => old(x)
  }
}
15
Wesley Miao

Wesley Milanoの回答に加えて、誰かが非推奨の警告について疑問に思っている場合に備えて、コードをsbt-Assemblyプラグインの新しいバージョン（つまり0.13.0）に少し適合させる必要があります。

assemblyMergeStrategy in Assembly := {
    case PathList("javax", "servlet", xs @ _*) => MergeStrategy.last
    case PathList("javax", "activation", xs @ _*) => MergeStrategy.last
    case PathList("org", "Apache", xs @ _*) => MergeStrategy.last
    case PathList("com", "google", xs @ _*) => MergeStrategy.last
    case PathList("com", "esotericsoftware", xs @ _*) => MergeStrategy.last
    case PathList("com", "codahale", xs @ _*) => MergeStrategy.last
    case PathList("com", "yammer", xs @ _*) => MergeStrategy.last
    case "about.html" => MergeStrategy.rename
    case "META-INF/ECLIPSEF.RSA" => MergeStrategy.last
    case "META-INF/mailcap" => MergeStrategy.last
    case "META-INF/mimetypes.default" => MergeStrategy.last
    case "plugin.properties" => MergeStrategy.last
    case "log4j.properties" => MergeStrategy.last
    case x =>
        val oldStrategy = (assemblyMergeStrategy in Assembly).value
        oldStrategy(x)
}
24
PermaFrost