1

I'm trying to access my S3 bucket with Hadoop (2.7.3) and I'm getting the following

ubuntu@AWS:~/Prototype/hadoop$ ubuntu@AWS:~/Prototype/hadoop$ bin/hadoop fs -ls s3://[bucket]/

17/03/24 15:33:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable -ls: Fatal internal error com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: 1FA2318A386330C0, AWS Error Code: null, AWS Error Message: Bad Request, S3 Extended Request ID: 1S7Eq6s9YxUb9bPwyHP73clJvD619LZ2o0jE8VklMAA9jrKXPbvT7CG6nh0zeuluGrzybiPbgRQ= at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031) at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994) at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325) at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235) at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218) at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201) at org.apache.hadoop.fs.shell.Command.run(Command.java:165) at org.apache.hadoop.fs.FsShell.run(FsShell.java:287) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.fs.FsShell.main(FsShell.java:340) ubuntu@AWS:~/Prototype/hadoop$

conf-site.xml:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>s3://[ Bucket ]</value>
    </property>

    <property>
            <name>fs.s3a.endpoint</name>
            <value>s3.eu-central-1.amazonaws.com</value>
    </property>

    <property>
        <name>fs.s3a.access.key</name>
        <value>[ Access Key Id ]</value>
    </property>

    <property>
        <name>fs.s3a.secret.key</name>
        <value>[ Secret Access Key ]</value>
    </property>

    <property>
        <name>fs.s3.awsAccessKeyId</name>
        <value>[ Access Key Id ]</value>
    </property>

    <property>
        <name>fs.s3.awsSecretAccessKey</name>
        <value>[ Secret Access Key ]</value>
    </property>

    <property>
        <name>fs.s3n.awsAccessKeyId</name>
        <value>[ Access Key Id ]</value>
    </property>

    <property>
        <name>fs.s3n.awsSecretAccessKey</name>
        <value>[ Secret Access Key ]</value>
    </property>

    <property>
        <name>fs.s3.impl</name>
        <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
    </property>

    <!-- Comma separated list of local directories used to buffer
         large results prior to transmitting them to S3. -->
    <property>
        <name>fs.s3.buffer.dir</name>
        <value>/tmp</value>
    </property>
</configuration>

Anyone knows what's the issue?

Edit: The bucket and the VMs accessing it are in Frankfurt. It seemed similar to https://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.8.0/bk_hdcloud-aws/content/s3-trouble/index.html but after adding the endpoint still it doesn't work.

razvan
  • 559
  • 7
  • 23
  • Its quite difficult to say from the error. Can you try granting anonymous access to the bucket temporally to see whether its a bucket access control issue? – Ashan Mar 25 '17 at 13:59
  • Granted permissions to everyone still the same error. – razvan Mar 27 '17 at 08:57

1 Answers1

3

Sounds like the V4 auth problem, which the fs.s3a.endpoint property should have fixed that

Clock problems can cause issues too. Check Joda time, and make sure that all your machines have caught up with this weekend's clock change.

Try also grabbing the Hadoop 2.8.0 RC3 and see if the problem has gone away then. If it is still there, that's the version to ask for help with on the apache lists.

stevel
  • 12,567
  • 1
  • 39
  • 50
  • Works with 2.8.0. Is it really RC (not final)? https://archive.apache.org/dist/hadoop/common/hadoop-2.8.0/ – razvan Mar 27 '17 at 13:57
  • I think the vote has gone through now, but I was answering yesterday & there were still problems with the upgrad, or more specifically getting a file that size and the signatures up – stevel Mar 28 '17 at 09:41