java.lang.IllegalAccessError Error while reading AWS S3 configuration from Java

Question

I am getting below error when trying to access the configuration from Java.

Exception in thread "main" java.lang.IllegalAccessError: tried to access method org.apache.hadoop.metrics2.lib.MutableCounterLong.<init>(Lorg/apache/hadoop/metrics2/MetricsInfo;J)V from class org.apache.hadoop.fs.s3a.S3AInstrumentation
    at org.apache.hadoop.fs.s3a.S3AInstrumentation.streamCounter(S3AInstrumentation.java:164)
    at org.apache.hadoop.fs.s3a.S3AInstrumentation.streamCounter(S3AInstrumentation.java:186)
    at org.apache.hadoop.fs.s3a.S3AInstrumentation.<init>(S3AInstrumentation.java:113)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:199)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
    at MyProgram.GetHiveTableData(MyProgram.java:710)
    at MyProgram$1.run(MyProgram.java:674)
    at MyProgram$1.run(MyProgram.java:670)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at MyProgram.GetHiveTableDetails(MyProgram.java:670)
    at MyProgram.main(MyProgram.java:398)

The Code line is

FileSystem hdfs = FileSystem.get(new URI(uriStr), configuration);

uriStr=s3a://sBucketName

Confurations are set as below for S3A

fs.default.name=fs.defaultFS
fs.defaultFS=s3a://bucketName
sPath: XXXXXX
fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
fs.s3a.access.key=XXXXXX
fs.s3a.secret.key=XXXXXXX
fs.s3a.endpoint=XXXXXXX
hadoop.rpc.protection=privacy
dfs.data.transfer.protection=privacy
hadoop.security.authentication=Kerberos
dfs.namenode.kerberos.principal=hdfs/XXXX@XXXX.XXX.XXXXXX.XXX
yarn.resourcemanager.principal=yarn/XXXX@XXXX.XXX.XXXXXX.XXX

Am I missing anything in configuration setup? Please advise.

Could you please post the answer here? It would be really helpful. — Giselle Van Dongen, Aug 06 '18 at 11:56
Could you please post the solution here . It would be helpful . Thanks — Balaji V, Sep 25 '18 at 09:36

buxizhizhoum · Answer 1 · 2018-11-13T01:39:06.043

This problem might occurs if the version of aws-sdk and hadoop version is not compatible, you may get more help from Spark job reading from S3 on Spark cluster gives IllegalAccessError: tried to access method MutableCounterLong and java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics

When I roll back hadoop-aws version from 2.8.0 to 2.7.3 the problem is solved.

spark-submit --master local \
--packages org.apache.hadoop:hadoop-aws:2.7.3,\
com.amazonaws:aws-java-sdk-pom:1.10.6,\
org.apache.hadoop:hadoop-common:2.7.3 \
test_s3.py

According to discussion here https://stackoverflow.com/a/52828978/8025086, it seems it is proper to use aws-java-sdk1.7.4, I just tested this simple example with pyspark, it also works. I am not a java guy, may be someone could have a better explanation.

# this one also works, notice that the version of aws-java-sdk is different
spark-submit --master local \
--packages org.apache.hadoop:hadoop-aws:2.7.3,\
com.amazonaws:aws-java-sdk:1.7.4,\
org.apache.hadoop:hadoop-common:2.7.3 \
test_s3.py

it's mixing the hadoop- jar versions which is causing this specific problem (they are designed to be used in sync), but as the AWS SDK is also a moving target, you should stick with whatever hadoop was built with. 1.7.4 is what Hadoop 2.7.x came with, and is the only one safe to use. 1.10.x? Maybe, but its jackson dependency changed, so you've got new problems. See [https://issues.apache.org/jira/browse/HADOOP-13050](https://issues.apache.org/jira/browse/HADOOP-13050) — stevel, Nov 13 '18 at 14:53

java.lang.IllegalAccessError Error while reading AWS S3 configuration from Java

1 Answers1