I have a file transfer service written in Java and Spring boot where it streams files from source to destination without any local storage.
It gets handles of InputStream from source and OutputStream for destination and effectively does following to stream the file.
IOUtils.copy(inputStream, outputStream);
Destination InputStream -> IOUtils.copy -> Destination OutputStream
Now I have another requirement where I need to call an Encryption API which accepts an application/oct-ect stream and returns the same. The API encrypt the data as the client feed the data and returns the data as stream at the same time. I have tested the API with curl using following command:
curl -v -X POST -H "Content-Type:application/octet-stream" -H "Accept:application/octet-stream" --data-binary @test-file -o test-file.enc http://127.0.0.1:1234/encrypt/test-file
Now I need to integrate this Encryption API with my original File Transfer Service where I can get handle of the source InputStream feed it to the API request, receive the InputStream from the API response and plugin this in my original IOUtils.copy call to transfer the data to the destination:
Destination InputStream -> Encryption API -> Encrypted InputStream -> IOUtils.copy -> Destination OutputStream
To implement the encryption API call initially I am trying to build a client and feed a FileInputStream and extract the response into a FileOutputStream. I have tried both Apache HTTPClient and Spring RestTemplate. In both cases with smaller files it is working as expected however with large files the Encryption API is only receiving part of the data and it is seem to be stuck there. Original caller (File Transfer Service) is not getting any response back and hanging as well. The behaviour is identical in both implementations.
Implementation with HTTPClient:
InputStream unencryptedInputStream = new FileInputStream("test-file");
HttpClient httpclient = HttpClients.createDefault();
URIBuilder builder = new URIBuilder("http://127.0.0.1:1234/encrypt/test-file");
HttpPost request = new HttpPost(builder.build());
request.setHeader("Content-Type", "application/octet-stream");
InputStreamEntity inputStreamEntity = new InputStreamEntity(unencryptedInputStream, 1064041488);
inputStreamEntity.setContentType("application/octet-stream");
inputStreamEntity.setChunked(true);
request.setEntity(inputStreamEntity );
HttpResponse response = httpclient.execute(request);
HttpEntity entity = response.getEntity();
InputStream encryptedInputStream = entity.getContent();
OutputStream os = new FileOutputStream("test-file.enc");
IOUtils.copy(encryptedInputStream, os);
Implementation with Spring RestTemplate:
RestTemplate restTemplate = new RestTemplate();
InputStream unencryptedInputStream = new FileInputStream("test-file");
OutputStream encryptedOutputStream = new FileOutputStream("test-file.enc");
SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
requestFactory.setBufferRequestBody(false);
restTemplate.setRequestFactory(requestFactory);
RequestCallback requestCallback = new RequestCallback() {
@Override
public void doWithRequest(ClientHttpRequest clientHttpRequest) throws IOException {
clientHttpRequest.getHeaders().add("Content-type", "application/octet-stream");
clientHttpRequest.getHeaders().add("Accept", "application/octet-stream");
IOUtils.copy(unencryptedInputStream, clientHttpRequest.getBody());
}
};
ResponseExtractor<InputStream> responseExtractor = new ResponseExtractor<InputStream>() {
@Override
public InputStream extractData(ClientHttpResponse clientHttpResponse) throws IOException {
IOUtils.copy(clientHttpResponse.getBody(), encryptedOutputStream);
return clientHttpResponse.getBody();
}
};
restTemplate.execute("http://127.0.0.1:1234/encrypt/test-file", HttpMethod.POST, requestCallback, responseExtractor);
Both cases are working with small files but hangs on large files. I have tried to wrap the request IOUtils.copy in a 'Runnable' Thread but still the same issue.