spark Not able to read from s3

Question

I am creating my spark-shell using the below command.

spark-shell --packages org.apache.hadoop:hadoop-aws:3.1.1,com.amazonaws:aws-java-sdk-pom:1.11.392,org.wso2.orbit.joda-time:joda-time:2.9.4.wso2v1

Then I am running the below code to access a file in S3.

val accessKeyId = "myid"
val secretAccessKey = "mykey"
sc.hadoopConfiguration.set("fs.s3a.awsAccessKeyId", accessKeyId)
sc.hadoopConfiguration.set("fs.s3a.awsSecretAccessKey",secretAccessKey)
sc.hadoopConfiguration.set("fs.s3a.impl","org.apache.hadoop.fs.s3a.S3AFileSystem")
val lines = sc.textFile("s3a://bucket-name/path-to-file")

Now runnning the below code gives me the below error.

scala> lines.count()
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StreamCapabilities
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
  at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

Possible duplicate of [java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V](https://stackoverflow.com/questions/49158612/java-lang-nosuchmethoderror-org-apache-hadoop-conf-configuration-reloadexisting) — stevel, Aug 26 '18 at 12:18

score 1 · Accepted Answer · answered Aug 22 '18 at 06:20

1

Unable to find dependency of org.apache.hadoop.fs.StreamCapabilities. Use

hadoop-common-3.1.jar

answered Aug 22 '18 at 06:20

Kishore

5,761
5
28
53

Now I am getting a different error. `java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V` – joydeep bhattacharjee Aug 22 '18 at 06:49
check this https://stackoverflow.com/questions/49158612/java-lang-nosuchmethoderror-org-apache-hadoop-conf-configuration-reloadexisting It is related to some version problem – Kishore Aug 22 '18 at 06:59

score 0 · Answer 2 · answered Aug 26 '18 at 11:58

Your stack traces are exactly what happens when you try to mix hadoop-* JARs. you can't do that. All you will do is move stack traces around.

This is covered in the latest s3a docs

ps: those aren't the property names for the s3a login details. Read that hadoop documentation

spark Not able to read from s3

2 Answers2