0

I have a file transfer service written in Java and Spring boot where it streams files from source to destination without any local storage.

It gets handles of InputStream from source and OutputStream for destination and effectively does following to stream the file.

IOUtils.copy(inputStream, outputStream);
Destination InputStream -> IOUtils.copy -> Destination OutputStream

Now I have another requirement where I need to call an Encryption API which accepts an application/oct-ect stream and returns the same. The API encrypt the data as the client feed the data and returns the data as stream at the same time. I have tested the API with curl using following command:

curl -v -X POST -H "Content-Type:application/octet-stream" -H "Accept:application/octet-stream" --data-binary @test-file -o test-file.enc http://127.0.0.1:1234/encrypt/test-file

Now I need to integrate this Encryption API with my original File Transfer Service where I can get handle of the source InputStream feed it to the API request, receive the InputStream from the API response and plugin this in my original IOUtils.copy call to transfer the data to the destination:

Destination InputStream -> Encryption API -> Encrypted InputStream -> IOUtils.copy -> Destination OutputStream

To implement the encryption API call initially I am trying to build a client and feed a FileInputStream and extract the response into a FileOutputStream. I have tried both Apache HTTPClient and Spring RestTemplate. In both cases with smaller files it is working as expected however with large files the Encryption API is only receiving part of the data and it is seem to be stuck there. Original caller (File Transfer Service) is not getting any response back and hanging as well. The behaviour is identical in both implementations.

Implementation with HTTPClient:

InputStream unencryptedInputStream = new FileInputStream("test-file");
HttpClient httpclient = HttpClients.createDefault();

URIBuilder builder = new URIBuilder("http://127.0.0.1:1234/encrypt/test-file");

HttpPost request = new HttpPost(builder.build());
request.setHeader("Content-Type", "application/octet-stream");

InputStreamEntity inputStreamEntity = new InputStreamEntity(unencryptedInputStream, 1064041488);
inputStreamEntity.setContentType("application/octet-stream");
inputStreamEntity.setChunked(true);

request.setEntity(inputStreamEntity );

HttpResponse response = httpclient.execute(request);
HttpEntity entity = response.getEntity();

InputStream encryptedInputStream = entity.getContent();

OutputStream os = new FileOutputStream("test-file.enc");
IOUtils.copy(encryptedInputStream, os);

Implementation with Spring RestTemplate:

RestTemplate restTemplate = new RestTemplate();

InputStream unencryptedInputStream = new FileInputStream("test-file");

OutputStream encryptedOutputStream = new FileOutputStream("test-file.enc");

SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
restTemplate.setRequestFactory(requestFactory);

RequestCallback requestCallback = new RequestCallback() {
    @Override
    public void doWithRequest(ClientHttpRequest clientHttpRequest) throws IOException {
        clientHttpRequest.getHeaders().add("Content-type", "application/octet-stream");
        clientHttpRequest.getHeaders().add("Accept", "application/octet-stream");

        IOUtils.copy(unencryptedInputStream, clientHttpRequest.getBody());
    }
};

ResponseExtractor<InputStream> responseExtractor = new ResponseExtractor<InputStream>() {
    @Override
    public InputStream extractData(ClientHttpResponse clientHttpResponse) throws IOException {
        IOUtils.copy(clientHttpResponse.getBody(), encryptedOutputStream);
        return clientHttpResponse.getBody();
    }
};

restTemplate.execute("http://127.0.0.1:1234/encrypt/test-file", HttpMethod.POST, requestCallback, responseExtractor);

Both cases are working with small files but hangs on large files. I have tried to wrap the request IOUtils.copy in a 'Runnable' Thread but still the same issue.

adesai
  • 370
  • 3
  • 22
  • How much memory is your app allowed to use? You effectively have to store 2 copies of the file in memory. It is possible to make it more efficient but not necessarily easily. I wouldn't be surprised if you are out of ram. – DCTID Aug 21 '20 at 01:47
  • should have mentioned, we have our own version of IOUtils.copy where it flushes the buffer. I have tested 2 GB files without encryption and no performance issue. – adesai Aug 21 '20 at 08:30

0 Answers0