1

I'm trying to find efficient way of downloading files from s3, insert them into zip file, and after inserting those files into archive, upload it to the bucket. It should be efficient as archive could reach almost 10Gb.I tried to use one thread with Zip4J`s library ZipOutputStream that is created from PipedOutputStream for inserting files into zip and simultaneously using another thread for uploading that zip to the bucket. The problem is result zip is corrupted, and I'm not able to copy any file from that zip, here's my code:

Part that download and insert files into zip archive:

private Thread getDownloadAndZipThread(List<String> keys, String filename, ZipFile zip, ZipParameters parameters) {
        return new Thread(() -> {
            long start = System.currentTimeMillis();
            try (final ZipOutputStream zipOutputStream = new ZipOutputStream(pipedOutputStream)){
                keys.forEach(key -> {
                    //logger.log("\nprocessing file: " + key);
                    try {
                        parameters.setFileNameInZip(key);
                        zipOutputStream.putNextEntry(parameters);
                        pipedOutputStream.write(s3Service.getFile(key,               BUCKET_NAME).readAllBytes());  
                        zipOutputStream.closeEntry();
                        //zip.addStream(IOUtils.copy(s3Service.getFile(key, BUCKET_NAME),  pipedOutputStream), parameters);
                    } catch (IOException e) {
                        throw new RuntimeException(e);
                    }
                });
            } catch (Exception e) {
                logger.log("\nZipService - getDownloadAndZipThread - error: " + e);
            }
            long executedTime = System.currentTimeMillis() - start;
            logger.log("\nZipService - getDownloadAndZipThread - execution time: " + executedTime);
        });
    }

Part that upload zip to the bucket:

private Thread getS3Out(String filename) {
    return new Thread(() -> {
        long start = System.currentTimeMillis();
        try {
            s3Service.multipartUpload(filename, BUCKET_NAME, pipedInputStream);
            pipedInputStream.close();
        } catch (final Exception all) {
            logger.log("\nFailed to process outputStream due to error: " + all + "\n");
            System.exit(-3);
        }
        long executedTime = System.currentTimeMillis() - start;
        logger.log("\nZipService - getS3Out - execution time: " + executedTime);
        });
}

Part that call these threads:

String filename = generateZipName();
 ZipFile zip = new ZipFile(filename);

        
 ZipParameters parameters = new ZipParameters();
 parameters.setCompressionLevel(CompressionLevel.FASTEST);

 final Thread downloadAndZip = getDownloadAndZipThread(keys, filename, zip, parameters); 
 final Thread upload = getS3Out(filename);

 downloadAndZip.start();
 upload.start();
 try {
      downloadAndZip.join();
      upload.join();
     } catch (InterruptedException e) {
      logger.log("ZipService - failed to join thread due to error: " + e);
      throw new RuntimeException(e);
    }

I tried using built-in Java ZipOutput stream, but it works too slow and as I need to use this code in AWS lambda it should be better. It take 130 seconds to download - insert into zip - upload to the bucket.

Zip4J ZipOutputStream instead takes 80 seconds in with the same cycle.

Both variants was tested on 16 files with total size 1.6Gb.

I need to somehow fix issue with corrupted zip file for Zip4j case.

Liubomur Pryimal
  • 11
  • 1
  • Writing directly to the `pipedOutputStream` backing your `zipOutputStream` sounds like a recipe for file corruption, yes. Also, instead of downloading the entire file from S3 with `readAllBytes`, consider reading smaller chunks from S3 and immediately writing those chunks back to your zip file. – Botje Jan 27 '23 at 13:00
  • 2
    Note: there are GitHub projects that do this type of thing e.g. https://github.com/DanielHindi/aws-s3-zipper. Not necessarily in Java, but might be helpful to you. – jarmod Jan 27 '23 at 13:37
  • @Botje thank you, really, writing data to `zipOutputStream` fixed the issue with corrupted zip =). I'm not sure what should I use instead of `readAllBytes`. Do you mean using `IOUtils.copy`? – Liubomur Pryimal Jan 29 '23 at 09:21
  • @LiubomurPryimal does the issue fixed? How you resolved? Is it working fine with 100gb of files? – codeSeeker May 15 '23 at 08:03

0 Answers0