web-dev-qa-db-ja.com

aws：EMRクラスターは、ジョブの送信時に「エラーUserData：ユーザーデータの取得中にエラーが発生しました」に失敗しますsparkジョブ

Aws EMRクラスターは正常に起動しましたが、送信は次のエラーで失敗します：

19/07/30 08:37:42 ERROR UserData: Error encountered while try to get user data
Java.io.IOException: File '/var/aws/emr/userData.json' cannot be read
    at com.Amazon.ws.emr.hadoop.fs.shaded.org.Apache.commons.io.FileUtils.openInputStream(FileUtils.Java:296)
    at com.Amazon.ws.emr.hadoop.fs.shaded.org.Apache.commons.io.FileUtils.readFileToString(FileUtils.Java:1711)
    at com.Amazon.ws.emr.hadoop.fs.shaded.org.Apache.commons.io.FileUtils.readFileToString(FileUtils.Java:1748)
    at com.Amazon.ws.emr.hadoop.fs.util.UserData.getUserData(UserData.Java:62)
    at com.Amazon.ws.emr.hadoop.fs.util.UserData.<init>(UserData.Java:39)
    at com.Amazon.ws.emr.hadoop.fs.util.UserData.ofDefaultResourceLocations(UserData.Java:52)
    at com.Amazon.ws.emr.hadoop.fs.util.AWSSessionCredentialsProviderFactory.buildSTSClient(AWSSessionCredentialsProviderFactory.Java:52)
    at com.Amazon.ws.emr.hadoop.fs.util.AWSSessionCredentialsProviderFactory.<clinit>(AWSSessionCredentialsProviderFactory.Java:17)
    at com.Amazon.ws.emr.hadoop.fs.rolemapping.DefaultS3CredentialsResolver.resolve(DefaultS3CredentialsResolver.Java:22)
    at com.Amazon.ws.emr.hadoop.fs.guice.CredentialsProviderOverrider.override(CredentialsProviderOverrider.Java:25)
    at com.Amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.executeOverriders(GlobalS3Executor.Java:130)
    at com.Amazon.ws.emr.hadoop.fs.s3.lite.executor.GlobalS3Executor.execute(GlobalS3Executor.Java:86)
    at com.Amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.invoke(AmazonS3LiteClient.Java:184)
    at com.Amazon.ws.emr.hadoop.fs.s3.lite.AmazonS3LiteClient.doesBucketExist(AmazonS3LiteClient.Java:90)
    at com.Amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.ensureBucketExists(Jets3tNativeFileSystemStore.Java:139)
    at com.Amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.initialize(Jets3tNativeFileSystemStore.Java:116)
    at com.Amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.initialize(S3NativeFileSystem.Java:508)
    at com.Amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.Java:111)
    at org.Apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.Java:2859)
    at org.Apache.hadoop.fs.FileSystem.access$200(FileSystem.Java:99)
    at org.Apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.Java:2896)
    at org.Apache.hadoop.fs.FileSystem$Cache.get(FileSystem.Java:2878)
    at org.Apache.hadoop.fs.FileSystem.get(FileSystem.Java:392)
    at org.Apache.spark.deploy.DependencyUtils$.org$Apache$spark$deploy$DependencyUtils$$resolveGlobPath(DependencyUtils.scala:190)
    at org.Apache.spark.deploy.DependencyUtils$$anonfun$resolveGlobPaths$2.apply(DependencyUtils.scala:146)
    at org.Apache.spark.deploy.DependencyUtils$$anonfun$resolveGlobPaths$2.apply(DependencyUtils.scala:144)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
    at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
    at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
    at org.Apache.spark.deploy.DependencyUtils$.resolveGlobPaths(DependencyUtils.scala:144)
    at org.Apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$3.apply(SparkSubmit.scala:354)
    at org.Apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$3.apply(SparkSubmit.scala:354)
    at scala.Option.map(Option.scala:146)
    at org.Apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:354)
    at org.Apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
    at org.Apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.Apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
    at org.Apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
    at org.Apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

serData.jsonは私のアプリケーションの一部ではなく、emr内部のようです。

何か間違っているアイデアはありますか？ livyリクエストを介してジョブを送信します。クラスターのセットアップ：2つのコアノードm4.large 7つのタスクノードm5.4xlarge 1つのマスターノードm5.xlarge

amazon-web-servicesapache-sparkamazon-emr
4
Normal

AWS EMR emr-5.24.1（spark 2.4.1）でも同様の問題に直面していますが、ジョブが失敗することはありません。

0
hnahak