httpbuilder-ng multipart/form-data with large application/zip part

Question

When trying to upload a large zip via multipart/form-data with httpbuilder-ng apache implementation version 0.16.1 i get

org.apache.http.ContentTooLongException: Content length is too long: 109370 at org.apache.http.entity.mime.MultipartFormEntity.getContent(MultipartFormEntity.java:103) at groovyx.net.http.ApacheEncoders.multipart(ApacheEncoders.java:74)

and that is Ok because the zip is rather large and there's no reason to wiggle it around buffers and ByteArray[Input|Output]Stream, question is how do i actually send the multipart to the connection's output stream? i tried to customize the encoder but the ToServer only exposes one method that accepts an InputStream which doesn't really work for me

Here is a snippet of what i was doing

configure {
        request.uri = 'https://anypoint.mulesoft.com'
        request.contentType = JSON[0]
        request.encoder(MULTIPART_FORMDATA[0], ApacheEncoders.&multipart)
    }
    .post {
        request.uri.path = '/cloudhub/api/v2/applications'
        request.headers['X-ANYPNT-ENV-ID'] = eid
        request.contentType = MULTIPART_FORMDATA[0]
        request.body = multipart {
            part('appInfoJson', JSON[0], '{"domain":"myDomain"}')
            part('autoStart', 'true')
            part(
                    'file',
                    'myLargeZip.zip',
                    BINARY[0], // or 'application/zip'
                    new File('/parent', 'myLargeZip.zip')
            )
        }
    }

I'm also curious if the `public interface ToServer { void toServer(InputStream inputStream); }` could be replaced/ reworked into exposing the connection's outputstream via Consumer — Vasile Glijin, Jun 12 '17 at 15:08
Your example seems correct - the error you are getting is coming from the apache client. You will need to modify the max allowed upload size somewhere on their client (you can configure it inside the HttpBuilder factory method before you return the client instance). I am not sure what you mean about where the set the stream contents... why would you need to? — cjstehno, Jun 12 '17 at 17:52
I did not think of increasing the max allowed upload size (***My bad***), but after poking around the Apache Client specifically `org.apache.http.entity.mime.MultipartFormEntity.getContent()` this method reads all the parts and loads them in to a ***ByteArrayInputStream*** this stream is passed to the ***ToServer*** which in this case would be an instance of ***ApacheToServer*** the stream is then read/transfered into a ***byte[]*** which i supposed is is written into the ***connection's OutputStream*** down the line. — Vasile Glijin, Jun 13 '17 at 10:13
I also noticed that ***ApacheToServer*** makes use of the ***HttpEntity#writeTo(OutputStream)***, i suspect the other implementations would do something similar as they need to pass the entity to the connection, now if ***ToServer*** were to accept a ***java.util.Consumer*** i'd be able to pass it the `org.apache.http.entity.mime.MultipartFormEntity.writeTo()` and skip the max upload size problem as well as 2 steps of transformation of InputStream into ByteArrayInputStream into byte[] which is then written into the connection's OutputStream. — Vasile Glijin, Jun 13 '17 at 10:28
Seems like upload size is not an issue for ***http-builder-ng-okhttp*** so for now i'll be using that. — Vasile Glijin, Jun 13 '17 at 10:36
I have created an issue related to documenting how large uploads should be handled... also one to investigate your suggestion for `ToServer`. Thanks. — cjstehno, Jun 13 '17 at 11:59
I was using http-build-ng-okhttp to upload a 1GB file using POST multipart. But there is still out of memory issue. Any streaming approach available for any of the existing implementation? — mengjiann, Mar 10 '20 at 02:50
@mengjiann have you found a way to solve the memory problem? — HansMaulwurf, May 18 '20 at 09:01
@HansMaulwurf sorry no. i ended up using another http client library for my implementation. — mengjiann, May 23 '20 at 03:09
@mengjiann alright, thx, would you mind telling me which lib? Thanks! — HansMaulwurf, May 25 '20 at 09:02
@HansMaulwurf im actually using the Apache HttpClient without the httpbuilder-ng. Here is the sample code: https://gist.github.com/mengjiann/a9140445d41cc05c5b71cb9e8e62028c if you need some reference. — mengjiann, May 26 '20 at 12:00

httpbuilder-ng multipart/form-data with large application/zip part

0 Answers0