REST Service - Multipart Post (JSON and File) as stream on https using Jersey

Question

I have a REST Service on https connection that accepts file upload as multipart (i.e. metadata of the file and file itself)

How can I use Jersey (for websphere) or HttpClient to call REST service and send file as multipart. I want send file as multiple streams of different sizes because we can have file more than 1GB. Moreover, the REST service is using Windows NT authentication for authorization and is on https.

Can anyone give example how I can achieve this? I have used multipart httpClient. Sending it as a stream does not work. Below is my code using httpClient 4.5.2

====================================

InputStream stream = new FileInputStream("test.doc");

MultipartEntityBuilder entityBuilder = MultipartEntityBuilder.create();
entityBuilder.setStrictMode();

InputStreamBody streamBody = new InputStreamBody(stream, docHandler.getFilename());

FormBodyPart filePart = FormBodyPartBuilder.create()
                .setName("Binary")
                .addField(Constants.RETRIEVAL_NAME_FIELD, "test.doc")               
                .addField("Content-Type",docHandler.getContentType())
                .setBody(streamBody)
                .build();
entityBuilder.addPart(filePart);

HttpPost httpPostRequest = new HttpPost();
httpPostRequest.setEntity(entityBuilder.build());

httpClient.execute(httpPostRequest);

==================================== But when I execute this code, I am getting following error

org.apache.http.client.NonRepeatableRequestException: Cannot retry request with a non-repeatable request entity

Any idea why I am getting this error. If I convert stream to byte array and use ByteArrayBody instead, then it works fine but I noticed in Fiddler that three request calls are being made to the server and in every call the entire content of the file is being copied. So, if my file is of 1GB then entire content will be sent to the server three times.

Firstly, how can I achieve sending large file in chunks or multiple streams so that entire file is not sent in one call. Secondly, is there a way to avoid having 3 calls to the server for NTLM authentication?

Any pointers?

Cheers!

Possible duplicate of [Jersey 2 Multipart upload Client](https://stackoverflow.com/questions/24637038/jersey-2-multipart-upload-client) — andih, Jun 06 '17 at 09:41
Hi andih, this link is using simple http connection and does not have authentication. In my case the RESTful service is on https and is using NTLM authentication mechanism. Is there any good example to upload file as a stream (I am inputstream) via REST service on https using NTLM authentication mechanism? Also, we have websphere where client java application will be running. Server is implemented in .Net. — sab123, Jun 11 '17 at 17:14
The Apache HTTP Client supports http and https. In most cases you don't even have to modify your code. NTLM authentication is something "on top" of http(s). Apache Http Clients Version > 4.1 (4.2.3) support [NTLM](https://hc.apache.org/httpcomponents-client-ga/ntlm.html) . It looks like you are asking different things within one question. How do chunk upload is a duplicate. NTLM Authentication scheme is described [here](https://www.innovation.ch/personal/ronald/ntlm.html). If you can avoid the three calls. The answer ist no. You should ask more specific questions. — andih, Jun 11 '17 at 18:13
andih, thanks for the response. Sorry for not answering the question clearly. — sab123, Jun 11 '17 at 18:36
andih. I am actually trying to find best way implement posting large files as a stream via REST service that uses NTLM authentication on https connection. I have tried it with httpclient4.5.2 and it works perfectly fine with https and NTLM authectication. But I am not able to do it as a stream. I have an inputStream and when I try posting using InputStreamBody I get error "stream is non-repeatable". Was able to do it using FileBody creating temp file but I do not want to create temp file. Is there a way to post file as a stream using httpClient4.5.2 on https using NTLM authentication. — sab123, Jun 11 '17 at 18:46

score 0 · Accepted Answer · answered Jun 12 '17 at 04:08

0

The Exception occurs because InputStreamBody is not repeatable (especially for large streams). A general InputStream can be read from only once.

FileBody is repeatable, as a File can be read from multiple times.

One reason for the repeatable reads may be the (NTLM)authorisation(did not check this).

You may avoid this by doing the first two NTML authorisation steps before the actual request and setting/sending the Authorization: NTLM <base64-encoded type-3-message> header, but that does not solve the problem, because the network may not be reliable enough and you have to retry anyway.

You basically have two options:

use repeatable ContentBody implementations only like FileBody or own repeatable ContentBody instance.
make sure the request does not need to be retried.

Please note the latter is not always possible. As mentioned before request retries due to authentication failures can be avoided, but those due to I/O errors cannot.

answered Jun 12 '17 at 04:08

andih

5,570
3
26
36

does it mean that we cannot post document as a stream using REST service with NTLM authentication or is it the limitation of httpClient? Is there any other REST Client API that can facilitate it like Jersey or any other? If posting as a stream is not possible as it is inherently non-repeatable, then where would InputStreamBody will be used as a multiple using httpClient? – sab123 Jun 12 '17 at 21:29
It has nothing to do with NTLM or client limitations. The client supports posting streams as soon as they are repeatable like the `FileBody` which is als sent as stream. The difference between `InputStream` and `File` is that with a `File` you can implement something like random access i.e. jump to a given point whereas with general `InputStreams` you can't. You can also overcome the authorisation stuff (not NTLM specific) by pre-authenicate / authorise the request but you still have to deal with IO errors. It's just if you want to use it this way you have to do some work. – andih Jun 13 '17 at 02:23
thanks Andih for the response. I thought filebody loads entire file in memory and then sends it as part of the request. If filebody is using stream internally then it should be fine then. We have an equivalent upload service which is on SOAP (basic authentication) to upload file and there are sending inputStream with MTOM enabled and it is working. In current scenario we have REST and was trying to figure out if inputStream can be sent to REST in the same fashion as we are doing in SOAP without creating a temp file. – sab123 Jun 13 '17 at 14:23
SOAP has the same io problems. I only know the MTOM examples using a `FileDataSource` but never used it. As already mentioned if you add the correct NTLM Authentication Header, which may get by leading HEAD, or GET request, it should be possible to use an `InputStreamBody`. If you want a robust solution you should go with the `FileBody` or your maybe your own (repeatable) `BufferedInputStreamBody` backed up by ?? (for example temp file). – andih Jun 13 '17 at 14:37

REST Service - Multipart Post (JSON and File) as stream on https using Jersey

1 Answers1