How to list all files from a specific S3 Bucket?

Question

I am using AWS S3 bucket for my project where I am uploading images and listing images using APIs, which is working pretty fine.

Now I want to list all files of a specific S3 bucket/folder (listing of objects of the specific bucket).

Here is the screenshot of my S3 bucket:

I tried to give bucket names like

1. wevieu/development/user_default/
2. wevieu/development/user_default
3. s3://wevieu/development/user_default/

etc. but nothing helped me.

Most of the time I am getting this error or any other error with 400 response:

The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey;

I tried 1 , 2, 3 solutions but didn't get help from any of this.

Help me if anyone have done this before successfully.

Note: I am using this SDK version com.amazonaws:aws-java-sdk-s3:1.11.423

Answer 1

Ok. So I got the solution.

You need to add the permissions before listing objects in your S3. If permissions are missing then your code will give you errors.

The code that I did use is this, which was not working before adding permission.

After adding permissions, I am able to list all objects.

objectListing = s3client.listObjects("wevieu", "development/user_default");
        if (objectListing != null) {
            List<S3ObjectSummary> s3ObjectSummariesList = objectListing.getObjectSummaries();
            if (!s3ObjectSummariesList.isEmpty()) {
                for (S3ObjectSummary objectSummary : s3ObjectSummariesList) {
                    System.out.println("file name:"+objectSummary.getKey());
                }
            }
        }

Answer 2

A better solution is to use the latest Amazon S3 V2 API as opposed to the old V1 API. Amazon strongly recommends moving to V2.

The AWS SDK for Java 2.x is a major rewrite of the version 1.x code base. It’s built on top of Java 8+ and adds several frequently requested features. These include support for non-blocking I/O and the ability to plug in a different HTTP implementation at run time.

Here is the solution using the S3 Java V2 API:

package com.example.s3;

// snippet-start:[s3.java2.list_objects.import]
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.ListObjectsRequest;
import software.amazon.awssdk.services.s3.model.ListObjectsResponse;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.model.S3Object;
import java.util.List;
import java.util.ListIterator;
// snippet-end:[s3.java2.list_objects.import]

/**
 * To run this AWS code example, ensure that you have setup your development environment, including your AWS credentials.
 *
 * For information, see this documentation topic:
 *
 * https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
 */

public class ListObjects {

    public static void main(String[] args) {

        final String USAGE = "\n" +
                "Usage:\n" +
                "    ListObjects <bucketName> \n\n" +
                "Where:\n" +
                "    bucketName - the Amazon S3 bucket from which objects are read. \n\n" ;

        if (args.length != 1) {
           System.out.println(USAGE);
           System.exit(1);
        }

        String bucketName = args[0];
        Region region = Region.US_WEST_2;
        S3Client s3 = S3Client.builder()
                .region(region)
                .build();

        listBucketObjects(s3, bucketName);
        s3.close();
    }

    // snippet-start:[s3.java2.list_objects.main]
    public static void listBucketObjects(S3Client s3, String bucketName ) {

       try {
            ListObjectsRequest listObjects = ListObjectsRequest
                    .builder()
                    .bucket(bucketName)
                    .build();

            ListObjectsResponse res = s3.listObjects(listObjects);
            List<S3Object> objects = res.contents();

            for (ListIterator iterVals = objects.listIterator(); iterVals.hasNext(); ) {
                S3Object myValue = (S3Object) iterVals.next();
                System.out.print("\n The name of the key is " + myValue.key());
                System.out.print("\n The object is " + calKb(myValue.size()) + " KBs");
                System.out.print("\n The owner is " + myValue.owner());

             }

        } catch (S3Exception e) {
            System.err.println(e.awsErrorDetails().errorMessage());
            System.exit(1);
        }
    }
    //convert bytes to kbs
    private static long calKb(Long val) {
        return val/1024;
    }
  

// snippet-end:[s3.java2.list_objects.main] }

You can find other Amazon S3 code examples here:

https://github.com/awsdocs/aws-doc-sdk-examples/tree/master/javav2/example_code/s3