1

I have a server with a restful-endpoint that consumes a JSON-object. I am able to send a JSON-object to the server with files up to 500MB, I am base64-encoding the files before I send them and packaging them in JSONObject (org.codehaus.jettison.json.JSONObject). Files up to a certain size works (ca 500MB). I would be able to send files up to 3 to 4 GB of size really.

If there is another way to package the file instead of base64 before wrapping it in a JSONObject I will take that advice (byte array?).

Environment: Running on Ubuntu18, memory 16G (Free 10G), Swap 16G. Openjdk 1.8.

I have tried a number of approaches.

  1. java.util.base64 (JDK8)
  2. org.apache.commons.codec.binary.Base64 (version 1.13)
  3. com.google.common.io.BaseEncoding (version 23.0)

The Error in the below sample is always on the row:

String filePath = "/tmp/1gb.zip";
System.out.println("\t Post to  : ".concat(URL));
System.out.println("\t file : ".concat(filePath));
Path path = Paths.get(filePath);
byte[] bArray = Files.readAllBytes(path);
byte[] encodeBase64 = Base64.encodeBase64(bArray);
final String contentToBeSaved = new String(encodeBase64);
HttpClient client = HttpClientBuilder.create().build();
HttpResponse response = null;
JSONObject metadata = new JSONObject();
metadata.put("owner", "Ingo");
metadata.put("access", "public");
metadata.put("licenseType", "CC BY");
metadata.put("fileName", "fileName");
metadata.put("fileDataBase64", contentToBeSaved);
String metadataFormatted = StringEscapeUtils.unescapeJavaScript(metadata.toString());
StringEntity entity = new StringEntity(metadataFormatted, ContentType.APPLICATION_JSON);
HttpPost post = new HttpPost(URL);
post.setEntity(entity);
response = client.execute(post);
HttpEntity responseEntity = response.getEntity();
String responseFromMediaserver = EntityUtils.toString(responseEntity, "UTF-8");
System.out.println("\n****");
System.out.println("Response is : " + responseFromMediaserver);
JSONObject json = new JSONObject(responseFromMediaserver);
String uuid = json.getString("uuid");
System.out.println("UUID is " + uuid);

Error:

Exception in thread "main" java.lang.NegativeArraySizeException at org.apache.commons.codec.binary.BaseNCodec.resizeBuffer(BaseNCodec.java:253) at org.apache.commons.codec.binary.BaseNCodec.ensureBufferSize(BaseNCodec.java:269)
 at org.apache.commons.codec.binary.Base64.encode(Base64.java:380)
 at org.apache.commons.codec.binary.BaseNCodec.encode(BaseNCodec.java:451)
 at org.apache.commons.codec.binary.BaseNCodec.encode(BaseNCodec.java:430)
 at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:679)
 at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:642)
 at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:623)
 at org.apache.commons.codec.binary.Base64.encodeBase64(Base64.java:556)
 at se.nrm.bio.mediaserver.testing.base64.NewClientEncodePost.posting(NewClientEncodePost.java:55)
 at se.nrm.bio.mediaserver.testing.base64.NewClientEncodePost.main(NewClientEncodePost.java:38)
InKi
  • 79
  • 2
  • 10
  • Do you really want to send it as JSON instead of just bytes? – Kayaman Nov 07 '19 at 11:54
  • I believe you would need to modify the file size limit in Apache. – Daniel Gale Nov 07 '19 at 11:58
  • 1
    While Base64 is able to handle such big files, it's not designed to do so. If you want to transfer files, you should probably look at other options like simple transfers of files over http or ftp, or whatever. Encoding in Base64 just for the sake of using JSON is not worth the trouble. – Olivier Grégoire Nov 07 '19 at 12:32
  • Thank you! The file ('contentToBeSaved') that I am sending is written to a directory (a zip-file) by another program, the first requirement was that the file would be <=100mb (that worked fine), the feature-request is that the file can be up to 2GB. I would like add some metadata, that is why I am packaging this in JSON. I have total control over the 'mediaserver'-sourcecode, which is the endpoint, so I could rewrite that endpoint... is it possible to fetch the zip-file and attach some metadata to it ( 2-3 attributes, in the header ?) and post the file to the mediaserver ? best, i – InKi Nov 07 '19 at 19:52
  • Sending this in a single message rather than as a stream might be problematic. Guava's `BaseEncoding` supports streaming, which might be useful. But honestly, I really think you should abandon sending the whole thing in one hunk of JSON. – Louis Wasserman Nov 07 '19 at 21:19
  • Thank you Louis for your advise. I created this example as an answer to your thoughts ( https://stackoverflow.com/questions/58809689/resteasy-and-the-multipartform-server-and-client ) posting a binary with some metadata using the jboss-jersey libraries - but I am running into some unexpected troubles ( trying to use @MultipartForm and a POJO server-side, but doing somthing wrong on the client-side) best, i – InKi Nov 11 '19 at 23:00
  • @DanielGale , I don't think that is the problem here - this solution is deployed in Tomcat. – InKi Nov 12 '19 at 07:36
  • Perhaps the same limit in the Tomcat configuration? https://tecadmin.net/increase-tomcat-upload-file-size-limit/ – Daniel Gale Nov 13 '19 at 11:48

0 Answers0