Error while copying external jars to /home/hadoop/lib folder EMR Amazon

Question

I am copying my external jars to /home/hadoop/lib directoy in EMR as a bootstrap process. But it is showing following error during bootstrap process

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager: method <init>()V not found
    at com.amazonaws.http.ConnectionManagerFactory.createThreadSafeClientConnManager(ConnectionManagerFactory.java:26)
    at com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:95)
    at com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:117)
    at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:60)
    at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:295)
    at org.apache.hadoop.fs.s3native.NativeS3FileSystem.initialize(NativeS3FileSystem.java:500)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1413)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:68)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1431)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:256)
    at com.amazon.elasticmapreduce.scriptrunner.ScriptRunner.fetchFile(ScriptRunner.java:39)
    at com.amazon.elasticmapreduce.scriptrunner.ScriptRunner.main(ScriptRunner.java:56)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:187)

It means that there are other dependency libraries required by your external jar, or there is version mismatch as described here :http://stackoverflow.com/questions/11103325/amazon-credentials-method-not-found — Amar, Mar 19 '14 at 13:28

score 1 · Answer 1 · answered Jan 14 '15 at 09:57

You need to do this as a bootstrap action. It's as simple as placing a script to do the copying into S3, and then if you're starting EMR from the command line, add a parameter like this:

--bootstrap-action 's3://my-bucket/boostrap.sh'

Or if you're doing it through the web interface, just enter the location of the file in as a "Custom action" in "Bootstrap Actions".

score 0 · Answer 2 · answered Jun 25 '14 at 10:55

I assume that you are using java process to download the external jars to /home/hadoop/lib/ and you are using amazon s3 java client to download the same.

The amazon java sdk for s3 needs other dependancy jars to be present when u run code .

You need to add them in correct order and version number. alternatively you can use shell script to download external jars to /home/hadoo/lib

Error while copying external jars to /home/hadoop/lib folder EMR Amazon

2 Answers2