8

I have an auto-configured AWS, Spring Boot application, and I'm trying to setup an endpoint that will simply download a particular file from a given bucket in Amazon S3. I uploaded a JPEG file into the bucket from my computer using the AWS console - now I'm trying to download that file using my Spring Boot API.

I'm getting the following error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied;

I have created a user and a group (user is in the group) on AWS console; the user/group has full access permissions on S3 as well as administrator access. I downloaded the access-key/secret-key pair and, for testing purposes, literally pasted the keys into my application.properties file as shown below (keys are not shown here, obviously :) ).

I'm confused as to why I'm still getting access denied. I've been searching and working on this for a while; I can't seem to find a solution to this issue that is specific to Spring Boot. Any help would be greatly appreciated.

application.properties:

cloud.aws.credentials.accessKey=myaccesskey
cloud.aws.credentials.secretKey=mysecretkey
cloud.aws.credentials.instanceProfile=false
cloud.aws.stack.auto=false

cloud.aws.region.auto=true
cloud.aws.region.static=myregion

SimpleResourceLoadingBean.java:

@RestController
public class SimpleResourceLoadingBean {

    private static Logger log = LoggerFactory.getLogger(HealthMonitorApplication.class);

    @Autowired
    private ResourceLoader resourceLoader;


    @RequestMapping("/getresource")
    public String resourceLoadingMethod() throws IOException {
        log.info("IN RESOURCE LOADER");

        Resource resource = this.resourceLoader.getResource("s3://s3.amazonaws.com/mybucket/myfile.ext");

        InputStream inputStream = resource.getInputStream();

        return inputStream.toString();
    }
}

pom.xml (Just the dependencies that are relevant to the question)

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-aws</artifactId>
            <version>1.1.0.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-aws-autoconfigure</artifactId>
            <version>1.1.0.RELEASE</version>
        </dependency>
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
corecase
  • 1,278
  • 5
  • 18
  • 29
  • Change cloud.aws.credentials.instanceProfile=false to true and check if it works? – Piyush Patil Jul 26 '16 at 00:04
  • Tried and failed :1 – corecase Jul 26 '16 at 00:15
  • Is the above Spring application on local or on a EC2 instance? – Piyush Patil Jul 26 '16 at 00:48
  • The application is local. – corecase Jul 26 '16 at 00:53
  • Can you paste what you have in this file ~/.aws/credentials without the credentials on you local. – Piyush Patil Jul 26 '16 at 00:57
  • Hmm, I don't actually have a directory called '.aws' locally. However, I have the credentials (access key and secret key) listed in my application.properties file, and I know that my application is properly using these credentials, because I tried using different (random) credentials and I received the appropriate errors when doing so (e.g Access Key ID does not exist, Secret Key does not match Access Key ID). I don't get any errors regarding my credentials when using the correct credentials. – corecase Jul 26 '16 at 01:31
  • I think there is an issue when `ResourceLoader` is being initialized. It requires the S3 client. Might be related to this bug: https://github.com/spring-cloud/spring-cloud-aws/issues/228 – alltej Feb 09 '18 at 17:04

3 Answers3

22

Figured out the solution. Besides the application.properties configuration, I had to create a configuration class that would give me access to an AmazonS3Client object when provided the appropriate credentials. I followed this example on GitHub:

https://github.com/brant-hwang/spring-cloud-aws-example/blob/master/src/main/java/com/axisj/spring/cloud/aws/AWSConfiguration.java

AWSConfiguration.java:

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3Client;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class AWSConfiguration {

    @Value("${cloud.aws.credentials.accessKey}")
    private String accessKey;

    @Value("${cloud.aws.credentials.secretKey}")
    private String secretKey;

    @Value("${cloud.aws.region}")
    private String region;

    @Bean
    public BasicAWSCredentials basicAWSCredentials() {
        return new BasicAWSCredentials(accessKey, secretKey);
    }

    @Bean
    public AmazonS3Client amazonS3Client(AWSCredentials awsCredentials) {
        AmazonS3Client amazonS3Client = new AmazonS3Client(awsCredentials);
        amazonS3Client.setRegion(Region.getRegion(Regions.fromName(region)));
        return amazonS3Client;
    }
}

Once this is configured, you can create AmazonS3Client objects (autowired) in your other classes, and use the client to make requests to your S3 cloud. The example uses a wrapper class as a service in order to ease the implementation of additional controller classes.

S3Wrapper.java:

import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.*;
import org.apache.commons.io.IOUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Service;
import org.springframework.util.StringUtils;
import org.springframework.web.multipart.MultipartFile;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

@Service
public class S3Wrapper {

    @Autowired
    private AmazonS3Client amazonS3Client;

    @Value("${cloud.aws.s3.bucket}")
    private String bucket;

    private PutObjectResult upload(String filePath, String uploadKey) throws FileNotFoundException {
        return upload(new FileInputStream(filePath), uploadKey);
    }

    private PutObjectResult upload(InputStream inputStream, String uploadKey) {
        PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, uploadKey, inputStream, new ObjectMetadata());

        putObjectRequest.setCannedAcl(CannedAccessControlList.PublicRead);

        PutObjectResult putObjectResult = amazonS3Client.putObject(putObjectRequest);

        IOUtils.closeQuietly(inputStream);

        return putObjectResult;
    }

    public List<PutObjectResult> upload(MultipartFile[] multipartFiles) {
        List<PutObjectResult> putObjectResults = new ArrayList<>();

        Arrays.stream(multipartFiles)
                .filter(multipartFile -> !StringUtils.isEmpty(multipartFile.getOriginalFilename()))
                .forEach(multipartFile -> {
                    try {
                        putObjectResults.add(upload(multipartFile.getInputStream(), multipartFile.getOriginalFilename()));
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                });

        return putObjectResults;
    }

    public ResponseEntity<byte[]> download(String key) throws IOException {
        GetObjectRequest getObjectRequest = new GetObjectRequest(bucket, key);

        S3Object s3Object = amazonS3Client.getObject(getObjectRequest);

        S3ObjectInputStream objectInputStream = s3Object.getObjectContent();

        byte[] bytes = IOUtils.toByteArray(objectInputStream);

        String fileName = URLEncoder.encode(key, "UTF-8").replaceAll("\\+", "%20");

        HttpHeaders httpHeaders = new HttpHeaders();
        httpHeaders.setContentType(MediaType.APPLICATION_OCTET_STREAM);
        httpHeaders.setContentLength(bytes.length);
        httpHeaders.setContentDispositionFormData("attachment", fileName);

        return new ResponseEntity<>(bytes, httpHeaders, HttpStatus.OK);
    }

    public List<S3ObjectSummary> list() {
        ObjectListing objectListing = amazonS3Client.listObjects(new ListObjectsRequest().withBucketName(bucket));

        List<S3ObjectSummary> s3ObjectSummaries = objectListing.getObjectSummaries();

        return s3ObjectSummaries;
    }
}

Note: The following dependency will need to be added to pom.xml in order to use the Apache Commons IO library.

pom.xml:

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-io</artifactId>
    <version>1.3.2</version>
</dependency>
corecase
  • 1,278
  • 5
  • 18
  • 29
  • But spring-boot provides auto configuration by adding EnableAutoConfiguration annotations to one of your Configuration classes., So why not configuring aws. – MV53 Dec 06 '16 at 07:18
  • When I download the file from this it is a binary file, not the png I was using – AlexK Apr 16 '18 at 20:33
  • This was quite a while ago, but you’d probably have to read the file as a png using an image reader or something like that if you’re trying to read it in your java code. If you’re just trying to open the file on your file system, I had no trouble with that; it literally downloads the file directly as it is to your file system (in whatever directory you’ve specified). – corecase Apr 16 '18 at 20:39
  • I cannot open the file on my system.. That is the precise issue. It is a strange binary file and there's not much I can do with it. – AlexK Apr 16 '18 at 21:05
  • Hmm, that is definitely strange... Wish I could be more helpful, but unfortunately I’m not too sure what the problem is here, because I never had any similar trouble as far as I can remember - as long as I was able to download the files, they were exactly the same as those in my S3 bucket. I would suggest opening up a new question on SO specifically regarding the issue you’re experiencing, so that someone else may be able to help you. – corecase Apr 16 '18 at 22:58
  • I don't know why. I did what you have done here. But, when I compile and run it. It give me an error that says "org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.cloud.aws.context.support.io.ResourceLoaderBeanPostProcessor#0': Cannot resolve reference to bean 'amazonS3' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'amazonS3': Cannot resolve reference to bean 'org.springframework.cloud.aws.core.region.RegionProvid.....;" – Rongeegee Apr 19 '19 at 06:36
  • I was o was looking for... but a have to downgrade the AWS SDK version (2.9.10)... :( But your answer helped alot! Thanks! – Milton Jacomini Neto Oct 18 '19 at 22:36
9

The accepted answer is using a deprecated APIs. Here's an updated revision.

First, update your maven dependencies:

 <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-java-sdk</artifactId>
        <version>1.11.274</version>
    </dependency>

AWSConfiguration.java

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class AWSConfiguration {

    @Value("${cloud.aws.credentials.accessKey}")
    private String accessKey;

    @Value("${cloud.aws.credentials.secretKey}")
    private String secretKey;

    @Value("${cloud.aws.region}")
    private String region;

    @Bean
    public BasicAWSCredentials basicAWSCredentials() {
        return new BasicAWSCredentials(accessKey, secretKey);
    }

    @Bean
    public AmazonS3 amazonS3Client(AWSCredentials awsCredentials) {
        AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard();
        builder.withCredentials(new AWSStaticCredentialsProvider(awsCredentials));
        builder.setRegion(region);
        AmazonS3 amazonS3 = builder.build();
        return amazonS3;
    }
}

S3Service.java

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.*;
import org.apache.commons.io.IOUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Service;
import org.springframework.util.StringUtils;
import org.springframework.web.multipart.MultipartFile;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

@Service
public class S3Service {

    @Autowired
    private AmazonS3 amazonS3;

    @Value("${cloud.aws.s3.bucket}")
    private String bucket;

    private PutObjectResult upload(String filePath, String uploadKey) throws FileNotFoundException {
        return upload(new FileInputStream(filePath), uploadKey);
    }

    private PutObjectResult upload(InputStream inputStream, String uploadKey) {
        PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, uploadKey, inputStream, new ObjectMetadata());

        putObjectRequest.setCannedAcl(CannedAccessControlList.PublicRead);

        PutObjectResult putObjectResult = amazonS3.putObject(putObjectRequest);

        IOUtils.closeQuietly(inputStream);

        return putObjectResult;
    }

    public List<PutObjectResult> upload(MultipartFile[] multipartFiles) {
        List<PutObjectResult> putObjectResults = new ArrayList<>();

        Arrays.stream(multipartFiles)
                .filter(multipartFile -> !StringUtils.isEmpty(multipartFile.getOriginalFilename()))
                .forEach(multipartFile -> {
                    try {
                        putObjectResults.add(upload(multipartFile.getInputStream(), multipartFile.getOriginalFilename()));
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                });

        return putObjectResults;
    }

    public ResponseEntity<byte[]> download(String key) throws IOException {
        GetObjectRequest getObjectRequest = new GetObjectRequest(bucket, key);

        S3Object s3Object = amazonS3.getObject(getObjectRequest);

        S3ObjectInputStream objectInputStream = s3Object.getObjectContent();

        byte[] bytes = IOUtils.toByteArray(objectInputStream);

        String fileName = URLEncoder.encode(key, "UTF-8").replaceAll("\\+", "%20");

        HttpHeaders httpHeaders = new HttpHeaders();
        httpHeaders.setContentType(MediaType.APPLICATION_OCTET_STREAM);
        httpHeaders.setContentLength(bytes.length);
        httpHeaders.setContentDispositionFormData("attachment", fileName);

        return new ResponseEntity<>(bytes, httpHeaders, HttpStatus.OK);
    }

    public List<S3ObjectSummary> list() {
        ObjectListing objectListing = amazonS3.listObjects(new ListObjectsRequest().withBucketName(bucket));

        List<S3ObjectSummary> s3ObjectSummaries = objectListing.getObjectSummaries();

        return s3ObjectSummaries;
    }
}
Mohamed Taher Alrefaie
  • 15,698
  • 9
  • 48
  • 66
  • When I download the file from this it is a binary file, not the png I was using. – AlexK Apr 16 '18 at 20:32
  • Don't you have to close the `s3Object` after reading the inputstream (in the `download` method)? – Wim Deblauwe May 02 '18 at 07:16
  • Why closeQuietly() is required after upload? And, is it required for any stream operation? I have two operation, one filePath is there, get file & upload, upload multipart file. Both case I am converting to input stream then calling putObject() common method. Is it required there also? – Satish Patro Mar 18 '21 at 06:36
1

Please correct url pattern

Resource resource = this.resourceLoader.getResource("s3://s3.amazonaws.com/mybucket/myfile.ext");

to

Resource resource = this.resourceLoader.getResource("s3://mybucket/myfile.ext");

As per documentation the pattern is s3://<bucket>/<object>.Its working in spring boot 2.0.6.RELEASE and spring cloud Finchley.SR2 (verified).

Reference : Spring Cloud AWS - Downloading files

Ashutosh
  • 2,827
  • 2
  • 12
  • 11
  • One doubt why to use resource loader when AmazonS3Client is available. Is it suitable for big file, too much load operation? – Satish Patro Mar 18 '21 at 07:21
  • 1
    Yes, one should use AmazonS3Client. For big files, I recommend using [TransferManager](https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html) – Ashutosh Mar 28 '21 at 08:59