Stream file from remote service to S3 bucket

Question

I have a microservice which needs to take a remote file, and upload it to a S3 bucket. The remote file is presented as a download link, and requires basic authentication.

Using the latest AWS 2.0 SDK I'm trying to stream the file so it doesn't have to all download to the server first. The files can be anywhere from 90kb -> 100GB+.

Using Spring Boot 2.0, I can't find if there is support in the new WebClient to handle this, so I'm trying to hack something together such as:

public Mono<String> upload(@PathVariable String projectId, @RequestBody String downloadLink) {
    try {
        String authString = properties.getSilverstripe().getUsername() + ":" + properties.getSilverstripe().getToken();

        // Download link needs to be cleaned a little; 
        URL url = new URL(downloadLink.replaceAll("\"", ""));
        URLConnection urlConnection = url.openConnection();

        // Add basic authentication to the stream
        urlConnection.setRequestProperty("Authorization", "Basic " + new String(Base64.encodeBase64(authString.getBytes())));
        InputStream inputStream = urlConnection.getInputStream();

        // Attempt to create a input stream and upload to the bucket
        BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
        Stream<String> stream = reader.lines();
        {
            stream.forEach(part -> {

                S3AsyncClient client = S3AsyncClient.create();
                client.putObject(
                        PutObjectRequest.builder()
                                .bucket(BUCKET)
                                .key(projectId + "/" + LocalDate.now() + ".sspak")
                                .build(),
                        AsyncRequestProvider.fromString(part)
                );

            });
        }

        return Mono.just(downloadLink);

    } catch (IOException e) {
        e.printStackTrace();
    }

    return null;
}

I was hoping there would be a pretty standard library/pattern for doing this but I can't find much online.

Any help at all is appreciated.

Not an answer, but a note: if you are uploading large files to S3 using the `S3Client` API, you should consider adding a `Content-Length` (and related) header(s). This allows better buffering of files while uploading, and without it the application would attempt to buffer the entire file in memory which is likely not what is desired (but potentially OK if file size is not large). — filpa, Jan 28 '18 at 22:49

Stream file from remote service to S3 bucket

0 Answers0