How do I upload a file to a pre-signed URL in AWS using Java?

Question

URL url = new URL("https://prod-us-west-2-uploads.s3-us-west-2.amazonaws.com/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A225178842088%3Aproject%3A1e6bbc52-5070-4505-b4aa-592d5e807b15/uploads/arn%3Aaws%3Adevicefarm%3Aus-west-2%3A225178842088%3Aupload%3A1e6bbc52-5070-4505-b4aa-592d5e807b15/501fdfee-877b-42b7-b180-de584309a082/Hamza-test-app.apk?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20181011T092801Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86400&X-Amz-Credential=AKIAJSORV74ENYFBITRQ%2F20181011%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=f041f2bf43eca1ba993fbf7185ad8bcb8eccec8429f2877bc32ab22a761fa2a");
        File file = new File("C:\\Users\\Hamza\\Desktop\\Hamza-test-app.apk");
        //Create Connection
        HttpURLConnection connection =  (HttpURLConnection) url.openConnection();
        connection.setDoOutput(true);
        connection.setRequestMethod("PUT");
        BufferedOutputStream bos = new BufferedOutputStream(connection.getOutputStream());
        BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
        int i;
        // read byte by byte until end of stream
        while ((i = bis.read()) > 0) {
    bos.write(i);
        }
        bos.flush();
        bis.close();
        bos.close();


        System.out.println("HTTP response code: " + connection.getResponseCode());
    }catch(Exception ex){
        System.out.println("Failed to Upload File");
    }

i want to upload a file to aws farm devices in java but file is not uploading to aws project upload list.

Answer 1

The simplest way is to create an Entity bypassing file

import org.apache.http.client.methods.HttpPut;
import org.apache.http.entity.mime.MultipartEntityBuilder;
import org.apache.http.entity.mime.content.FileBody;
import org.apache.http.impl.client.HttpClients;
import java.io.IOException;
import java.io.File;
public class Test {
    /**
     * Uploading file at pre-signed URL
     *
     * @throws IOException
     */
    private void uploadFileToAWSS3(String preSignedUrl) throws IOException {
        File file = new File("/Users/vmagadum/SitCopiedFile/temp/details.csv");
        HttpClient httpClient = HttpClients.custom()
                .setDefaultRequestConfig(
                        RequestConfig.custom().setCookieSpec(CookieSpecs.STANDARD).build()
                ).build();
        HttpPut put = new HttpPut(PRE_SIGNED_URL);
        HttpEntity entity = EntityBuilder.create()
                .setFile(file)
                .build();
        put.setEntity(entity);
        put.setHeader("Content-Type","text/csv");

        HttpResponse response = httpClient.execute(put);

        if (response.getStatusLine().getStatusCode() == 200) {
            System.out.println("File uploaded successfully at destination.");
        } else {
            System.out.println("Error occurred while uploading file.");
        }
    }
}

If you create an entity with MultipartEntityBuilder as below

HttpEntity entity = MultipartEntityBuilder.create()
                .addPart("file", new FileBody(file))
                .build();

Then it will add unnecessary data to the file. Here are the more details Link

Answer 2

Just to elaborate on my later comment here are two examples how to upload to the pre-signed URL returned by Device Farm's SDK in java.

Jenkins plugin example

Generic s3 documentation example about presigned urls

[update]

Here is an example which uploads a file to the Device Farm s3 presigned URL:

package com.jmp.stackoveflow;

import java.io.File;
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;

import com.amazonaws.ClientConfiguration;
import com.amazonaws.auth.AWSSessionCredentials;
import com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider;
import com.amazonaws.services.devicefarm.*;
import com.amazonaws.services.devicefarm.model.CreateUploadRequest;
import com.amazonaws.services.devicefarm.model.Upload;

import org.apache.commons.lang3.RandomStringUtils;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.entity.FileEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;

public class App {

    public static void main(String[] args) {
        String PROJECT_ARN = "arn:aws:devicefarm:us-west-2:111122223333:project:ffb3d9f2-3dd6-4ab8-93fd-bbb6be67b29b";
        String ROLE_ARN = "arn:aws:iam::111122223333:role/DeviceFarm_FULL_ACCESS";

        System.out.println("Creating credentials object");
        // gettting credentials
        STSAssumeRoleSessionCredentialsProvider sts = new STSAssumeRoleSessionCredentialsProvider.Builder(ROLE_ARN,
                RandomStringUtils.randomAlphanumeric(8)).build();
        AWSSessionCredentials creds = sts.getCredentials();
        ClientConfiguration clientConfiguration = new ClientConfiguration()
                .withUserAgent("AWS Device Farm - stackoverflow example");
        AWSDeviceFarmClient api = new AWSDeviceFarmClient(creds, clientConfiguration);
        api.setServiceNameIntern("devicefarm");

        System.out.println("Creating upload object");

        File app_debug_apk = new File(
                "PATH_TO_YOUR_FILE_HERE");
        FileEntity fileEntity = new FileEntity(app_debug_apk);

        CreateUploadRequest appUploadRequest = new CreateUploadRequest().withName(app_debug_apk.getName())
                .withProjectArn(PROJECT_ARN).withContentType("application/octet-stream").withType("ANDROID_APP");
        Upload upload = api.createUpload(appUploadRequest).getUpload();

        // Create the connection and use it to upload the new object using the
        // pre-signed URL.
        CloseableHttpClient httpClient = HttpClients.createSystem();
        HttpPut httpPut = new HttpPut(upload.getUrl());
        httpPut.setHeader("Content-Type", upload.getContentType());
        httpPut.setEntity(fileEntity);
        try {
            HttpResponse response = httpClient.execute(httpPut);
            System.out.println("Response: "+ response.getStatusLine().getStatusCode());
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
}

OUTPUT

Creating credentials object
Creating upload object
Response: 200

Answer 3

This is a bit of an old question. In case anyone else finds this. Here is how I solved the problem for files less than 5mb. For files over 5mb its recommended to use multi-part upload. NOTE: Using Java's "try with resources" is convenient. Try Catch makes this a clumsy operation but it ensures that resources are closed in the least amount of code within a method.

/**
 * Serial upload of an array of media files to S3 using a presignedUrl.
 */
public void serialPutMedia(ArrayList<String> signedUrls) {
        long getTime = System.currentTimeMillis();
        LOGGER.debug("serialPutMedia called");

        String toDiskDir = DirectoryMgr.getMediaPath('M');

        try {
            HttpURLConnection connection;
            for (int i = 0; i < signedUrls.size(); i++) {
                URL url = new URL(signedUrls.get(i));
                connection = (HttpURLConnection) url.openConnection();
                connection.setDoOutput(true);
                connection.setRequestMethod("PUT");
                localURL = toDiskDir + "/" + fileNames.get(i);
                try (BufferedInputStream bin = new BufferedInputStream(new FileInputStream(new File(localURL)));
                     ObjectOutputStream out = new ObjectOutputStream(new BufferedOutputStream(connection.getOutputStream())))  
                {
                    LOGGER.debug("S3put request built ... sending to s3...");
                    
                    byte[] readBuffArr = new byte[4096];
                    int readBytes = 0;
                    while ((readBytes = bin.read(readBuffArr)) >= 0) {
                        out.write(readBuffArr, 0, readBytes);
                    }
                    connection.getResponseCode();
                    LOGGER.debug("response code: {}", connection.getResponseCode());

                } catch (FileNotFoundException e) {
                    LOGGER.warn("\tFile Not Found exception");
                    LOGGER.warn(e.getMessage());
                    e.printStackTrace();
                }
            }

        } catch (MalformedURLException e) {
            LOGGER.warn(e.getMessage());
            e.printStackTrace();
        } catch (IOException e) {
            LOGGER.warn(e.getMessage());
            e.printStackTrace();
        }
        getTime = (System.currentTimeMillis() - getTime);
        System.out.print("Total get time in syncCloudMediaAction: {" + getTime + "} milliseconds, numElement: {" + signedUrls.size() + "}");
    }

Answer 4

These answers are all outdated as they are using AWS for Java V1. To perform this use case using best practice is to use AWS Java V2. Amazon Strongly recommends using V2 over V1.

Here is the Java V2 example that demonstrates how to use the S3Presigner client to create a presigned URL and upload an object to an Amazon Simple Storage Service (Amazon S3) bucket.

package com.example.s3;

// snippet-start:[presigned.java2.generatepresignedurl.import]
import java.io.IOException;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;
import java.time.Duration;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.presigner.model.PresignedPutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.PutObjectPresignRequest;
// snippet-end:[presigned.java2.generatepresignedurl.import]

/**
 * To run this AWS code example, ensure that you have setup your development environment, including your AWS credentials.
 *
 * For information, see this documentation topic:
 *
 * https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html
 */

public class GeneratePresignedUrlAndUploadObject {

    public static void main(String[] args) {

        final String USAGE = "\n" +
                "Usage:\n" +
                "    <bucketName> <keyName> \n\n" +
                "Where:\n" +
                "    bucketName - the name of the Amazon S3 bucket. \n\n" +
                "    keyName - a key name that represents a text file. \n" ;

        if (args.length != 2) {
            System.out.println(USAGE);
            System.exit(1);
        }

        String bucketName = args[0];
        String keyName = args[1];
        Region region = Region.US_EAST_1;
        S3Presigner presigner = S3Presigner.builder()
                .region(region)
                .build();

        signBucket(presigner, bucketName, keyName);
        presigner.close();
    }

    // snippet-start:[presigned.java2.generatepresignedurl.main]
    public static void signBucket(S3Presigner presigner, String bucketName, String keyName) {

        try {
            PutObjectRequest objectRequest = PutObjectRequest.builder()
                    .bucket(bucketName)
                    .key(keyName)
                    .contentType("text/plain")
                    .build();

            PutObjectPresignRequest presignRequest = PutObjectPresignRequest.builder()
                    .signatureDuration(Duration.ofMinutes(10))
                    .putObjectRequest(objectRequest)
                    .build();

            PresignedPutObjectRequest presignedRequest = presigner.presignPutObject(presignRequest);


            String myURL = presignedRequest.url().toString();
            System.out.println("Presigned URL to upload a file to: " +myURL);
            System.out.println("Which HTTP method needs to be used when uploading a file: " +
                    presignedRequest.httpRequest().method());

            // Upload content to the Amazon S3 bucket by using this URL
            URL url = presignedRequest.url();

            // Create the connection and use it to upload the new object by using the presigned URL
            HttpURLConnection connection = (HttpURLConnection) url.openConnection();
            connection.setDoOutput(true);
            connection.setRequestProperty("Content-Type","text/plain");
            connection.setRequestMethod("PUT");
            OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream());
            out.write("This text was uploaded as an object by using a presigned URL.");
            out.close();

            connection.getResponseCode();
            System.out.println("HTTP response code is " + connection.getResponseCode());

        } catch (S3Exception e) {
            e.getStackTrace();
        } catch (IOException e) {
            e.getStackTrace();
        }
    }
    // snippet-end:[presigned.java2.generatepresignedurl.main]
}

Answer 5

You should use the AWS SDK as shown here