Java heap out of memory issue while zipping list of nio paths

Question

I am asynchronously running the below method to zip the given set of nio paths. When there are multiple tasks running, java heap out of memory exception is encountered.

public InputStream compressToZip(String s3folderName, Set<Path> paths) throws Exception {
    try {
        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        ZipOutputStream zos = new ZipOutputStream(byteArrayOutputStream);
        paths.forEach(path -> {
            try {
                System.out.println("Zipping " + path.getFileName());
                zos.putNextEntry(new ZipEntry(path.getFileName().toString()));
                FileInputStream ObjectInputStream = new FileInputStream(path.toFile());
                IOUtils.copy(ObjectInputStream, zos);
                zos.closeEntry();
            } catch (Exception e) {
                ...
            }
        });
        zos.close();
        return new ByteArrayInputStream(byteArrayOutputStream.toByteArray());
    } catch (Exception e) {
        ...
    }
}

The input stream returned from this file will be written on sftp location.

org.springframework.integration.file.remote.session.Session session = this.createSession(deliveryType, deliveryLocation);    
zippedIpStream = fileCompressionSvc.compressToZip(s3folderName, fileDir);
session.write(zippedIpStream, deliveryLocation.getLocation().getFolder() + "/"
                                + getfileNameFormat(fileNameFormat, masterId) + ".zip");

I am not sure what went wrong to occur java heap issue. Could you please assist me.

You are writing the complete zip-file to memory, so your heap has to be larger than your maximum zip-file-size - and when used in parallel larger than the sum of all maximum zip-file-sizes... — piet.t, Oct 08 '21 at 08:41
As mentioned in the edit I am writing the same to a remote folder — mwKART, Oct 08 '21 at 09:03
@mwKART: yes, but **before** you do that, you write it all to an `ByteArrayOutputStream`. So instead of streaming directly to your remote folder, you have to hold everything in memory. You should 1. get an `OutputStream` to write to for the remote filesystem and then 2. create the `ZipOutputStream` by directly wrapping that instead of a `ByteArrayOutputStream`. — Joachim Sauer, Oct 08 '21 at 09:04
as per the code and this line `return new ByteArrayInputStream(byteArrayOutputStream.toByteArray());` everything is still in the memory. `ByteArrayOutputStream` will keep everything in memory until everything is written and flushed. So if the data is larger than your heap size you are eventually going to run into an OME. — Tanuj, Oct 08 '21 at 09:17
Thanks, All. the issue was `ByteArrayOutputStream`. More detail in the answer. — mwKART, Oct 08 '21 at 13:03

score 1 · Accepted Answer · answered Oct 08 '21 at 13:13

Changed the implementation to write the file into a file in local path and then send that file to sftp and then delete the temp zip file.

public void compressToZip(String s3folderName, Set<Path> distinctPaths, String efsPathWithFileName) throws Exception {
    try(FileOutputStream fos = new FileOutputStream(efsPathWithFileName);
        ZipOutputStream zos = new ZipOutputStream(fos)) {
        distinctPaths.forEach(path -> {
            try {
                zos.putNextEntry(new ZipEntry(path.getFileName().toString()));
                final FileInputStream fis = new FileInputStream(path.toFile());
                IOUtils.copy(fis, zos);
                zos.closeEntry();
            } catch (IOException e) {
                ...
            }
        });
    } catch (Exception e) {
        ...
        throw e;
    }
}

calling method:

InputStream zippedIpStream = new FileInputStream(tempCompressedFilePath);
session.write(zippedIpStream, deliveryLocation.getLocation().getFolder() + "/" + fileNameToWrite);
...
...                     
zippedIpStream.close();
...
...
Files.deleteIfExists(Paths.get(tempCompressedFilePath));

Java heap out of memory issue while zipping list of nio paths

1 Answers1