How to use the Amazon S3 Java API to perform getObject using an asynchronous call

Question

I need to download a lot of files from S3 so I wrapped many calls to s3Client.getObject(request, destination) with futures, but there's no mention of it being blocking so I'm afraid I might run into a race condition where all the download calls were made but the files weren't really downloaded yet.

Is it safe to assume this api is blocking since there's no mention of it being async?

To clarify, by blocking I mean that the flow won't continue after the s3Client.getObject(request, destination) line until the download is finished.

No, there is no 'blocking'. What would you imagine would cause blocking? — John Rotenstein, Sep 26 '21 at 08:22

score 1 · Accepted Answer · answered Sep 26 '21 at 09:00

1

Looking at the code on Github, it is indeed blocking the call (invoke).

answered Sep 26 '21 at 09:00

Thomas Jungblut

20,854
6
68
91

@smac2020 I assumed that the question was about the sync client and whether it is blocking - not about the async client which is obviously async. – Thomas Jungblut Sep 26 '21 at 18:21

score 1 · Answer 2 · answered Sep 26 '21 at 13:59

1

TO use the latest non-blocking code (AWS SDK for Java V2), refer to this Github example. This shows how to use getObject using the asynchronous client.

https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/javav2/example_code/s3/src/main/java/com/example/s3/S3AsyncStreamOps.java

answered Sep 26 '21 at 13:59

smac2020

9,637
4
24
38

good to know that there's an async client as well, thanks. – shinzou Oct 07 '21 at 09:19

score 0 · Answer 3 · answered Jul 20 '22 at 13:22

Using SDK V2 with reactive streams integration.

toBytes

Mono.fromFuture(s3Client.getObject(getObjectRequest, AsyncResponseTransformer.toBytes()))
          .map(ResponseBytes::asUtf8String)
          .flatMapMany(s -> Flux.fromArray(s.split("\n")))
          .map(stringLine -> ...)

toFile

Mono.fromFuture(s3Client.getObject(getObjectRequest, AsyncResponseTransformer.toFile(Path.of(localFilePath))))
          .thenMany(Flux.using(
                          // https://docs.oracle.com/javase/8/docs/api/java/nio/file/Files.html#lines-java.nio.file.Path-
                          // This method does not read all lines into a List, but instead populates lazily as the stream is consumed.
                          () -> Files.lines(Paths.get(localFilePath)),
                          Flux::fromStream,
                          Stream::close)
          .map(stringLine -> ...)

they are implementations of the `Publisher` interface in the reactive stream specification. https://projectreactor.io/docs/core/release/reference/#intro-reactive — Johan Perez, Jul 28 '22 at 09:25

How to use the Amazon S3 Java API to perform getObject using an asynchronous call

3 Answers3