AWS S3 file download via Java. Failing on Authentication call

Question

I'm trying to download a JSON file from an AWS S3 bucket through Java.

The file is created by a 3rd party billing application called Zuora.

The first step is to use OAuth credentials to generate the file. I then get a response with the file URL. I can access this via the browser and download it to my desktop, but when I try to process the file via Java I'm running into issues.

Everywhere I look online I see that people seem to have overcome similar issues by using AmazonS3Client from the AWS libraries. Ref: AWS S3 Java SDK - Download file help

I've tried this but as I'm accessing the bucket via a 3rd party I don't have the Auth ID & Secret to create credentials to make the call.

Do I need these credentials to do a GET call to a bucket on AWS S3? I'm thinking I shouldn't as I'm not trying to create a file or a bucket etc. I haven't been able to download the file via java without credentials.

Below I've added the URL that I'm trying to download the file from and the Java code.

Zuora response with URL: { "data": { "id": "fec47238-6a0f-48ef-9fb2-c7e24da886d5", "query": "select i.AccountId, i.Amount, i.PostedDate, i.InvoiceNumber, i.Status, i.Id from Invoice i, Subscription s where i.Status = 'Posted' and i.Amount > 0 and i.AccountId = s.AccountId and s.Id = '8ad084a67d58433a017d5cd0682c0b89'", "useIndexJoin": false, "sourceData": "LIVE", "queryStatus": "completed", "dataFile": "**https://bucket-name.s3.us-west-2.amazonaws.com/file-name.jsonl?X-Amz-Security-Token=some-tokenX-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20211215T110928Z&X-Amz-SignedHeaders=host&X-Amz-Expires=14400&X-Amz-Credential=Some-Credential&X-Amz-Signature=some-Signature**", "outputRows": 1, "processingTime": 1918, "remainingRetries": 3, "retries": 3, "updatedOn": "2021-12-15T11:09:28.123Z", "createdBy": "some-id" }

2 Java:

private final AmazonS3 amazonS3Client = AmazonS3ClientBuilder.standard().withRegion(Regions.US_WEST_2).build();

        try (final S3Object s3Object = amazonS3Client.getObject("bucket-name.s3.us-west-2.amazonaws.com",
                                                                "file-name.jsonl");
                final InputStreamReader streamReader = new InputStreamReader(s3Object.getObjectContent(), StandardCharsets.UTF_8);
                final BufferedReader reader = new BufferedReader(streamReader)) {
                System.out.println(reader.lines().collect(Collectors.toSet()));
        }

Error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;

Answer 1

To use a Service client exposed by an AWS SDK, you need creds. This is specified in an Developer guide - AWS SDK for Java 2.x:

https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html

Having said that - if the object in the S3 bucket has been presigned using the S3Presigner object and there is a presigned URL, you can access the object using a Java URI object without using creds (assuming you so do within the time period).

To learn more about presigned - see this topic in the DEV Guide:

Working with Amazon S3 presigned URLs

Here is a little Java SWING APP that can get content from an Amazon S3 bucket using a Presigned URL.

There is no need for creds here.

import javax.swing.*;
 import java.awt.*;
import java.awt.event.*;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;

class Swing implements ActionListener {
        JFrame frame=new JFrame();
        JButton button=new JButton("Click Me");

        Swing(){
            prepareGUI();
            buttonProperties();
        }

        public void prepareGUI(){
            frame.setTitle("My Window");
            frame.getContentPane().setLayout(null);
            frame.setVisible(true);
            frame.setBounds(200,200,400,400);
            frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        }
        public void buttonProperties(){
            button.setBounds(130,200,100,40);
            frame.add(button);
            button.addActionListener(this);
        }

        @Override
        public void actionPerformed(ActionEvent e) {
            //Get a presigned PDF from an Amazon S3 bucket.
            try {
                URL url  =  new URL("<Specify PRESIGNED URL> ") ;
                InputStream in = null;
                in = url.openStream();
                FileOutputStream fos = new FileOutputStream(new File("C:\\AWS\\yourFile2.txt"));
                System.out.println("reading from resource and writing to file...");
                int length = -1;
                byte[] buffer = new byte[1024];// buffer for portion of data from connection
                while ((length = in.read(buffer)) > -1) {
                    fos.write(buffer, 0, length);
                }
                fos.close();
                in.close();
                 System.out.println("File downloaded");
            } catch (IOException ex) {
                ex.printStackTrace();
            }
        }
    }

    public class HelloWorldSwing {
        public static void main(String[] args)
        {
            new Swing();
        }
    }