How to Reduce the time of GZIPOutputStream

Question

I have tried to gzip a large(100mb to 500mb) xml file.I have created method Zip to do that. the issues is that its talking too much time to zip.for 200mb it take 1.2 secs.i need to reduce the time too 100 millisecond for 100mb xml file. how do i optimize to reduce the time for zipping?

I have reduced the time by compromising little on compression ratio. tried another algorithms like Snappy,Lz4 but not much improvement and also they have poor compression.as per my knowledge gzipOutputStream.write() takes 85% of the time.so how can i optimize this step to get better performance with out compromising much of compression Ratio.

public static String zip(final String str) {
    if ((str == null) || (str.length() == 0)) {
        throw new IllegalArgumentException("Cannot zip null or empty string");
    }

    try (ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(str.length())) {
        try (GZIPOutputStream gzipOutputStream = new GZIPOutputStream(byteArrayOutputStream){{def.setLevel(Deflater.BEST_SPEED );}};) {
            gzipOutputStream.write(str.getBytes(StandardCharsets.UTF_8));

        } 
            T5 = System.currentTimeMillis();
            byte[] bytes=byteArrayOutputStream.toByteArray();
             T3 = System.currentTimeMillis();

            String zipped_text=DatatypeConverter.printBase64Binary(bytes);
             T4 = System.currentTimeMillis();
            return zipped_text;

    } catch(IOException e) {
        throw new RuntimeException("Failed to zip content", e);
    }

}

Answer 1

Here is my advice:

1. Create a proper benchmark so that you can get repeatable results. I would advise using a benchmarking framework; e.g. JMH.

2. Profile your code / benchmark to identify where the bottlenecks / hotspots are; e.g. using jVisualVM or Java Mission Control Flight Recorder.

3. Use the benchmarks and profiling results to guide your optimization effort.

(I would NOT rely simply on calls to System.currentTimeMillis() for a variety of reasons.)

One possible explanation is that a significant percentage of the time is spent on data copying in the following steps.

• Creating the input string containing the XML
• Capturing the compressed bytes in a ByteArrayOutputStream
• Concerting the bytes into another String.

So if you are looking for ways to improve this, try to arrange things so that the XML serializer writes to a pipeline that streams data through gzip and base64 conversion and then writes directly to a file or socket stream.

Also, I would avoid using base64 if possible. If the compressed XML is in an HTTP response, you should be able to send it in binary. It will be faster, and generate significantly less network traffic.

Finally, pick a compression algorithm that gives a good compromise between compression ratio and compression time.

---

How can I optimize this step to get better performance with out compromising the compression ratio.

If you are trying to do that, your goals are probably wrong. (And why did you then Base64 encode the compressed file? That contradicts your goal!)

---

Updates to address your comments:

1. You will (I think) get better performance by streaming than by turning your XML into a String and then calling getBytes() on it. For a start, the getBytes() call is making an unnecessary copy of the string content.

2. The Wikipedia page on Lossless Compression links to a number of algorithms, many of which should have readily available Java implementations. In addition, it has links to a number of benchmarks. I haven't looked at the benchmark links, but I expect at least one will quantify the compression versus compute time trade-off for different algorithms.

3. If you change the database table from CLOB to BLOB:

   • you can dispense with the base64, saving ~25% storage space
   • you can dispense with the base64 encoding step, saving a few percent of CPU
   • you can then pick a faster (but less compact) algorithm, saving more time at the cost of some of the space that you saved by going to a BLOB.

4. "I can't really change it its business requirement." - Really? If the database schema is a business requirement, then there is something really screwed up with your business. And on the flip-side, if the business is dictating the technology at that level, then they are also dictating the performance.

   There is no sound technical reason to store compress data as CLOBs.

5. As someone noted, the easiest way to get faster compression is to buy a faster computer. Or (my idea) a bank of computers so that you can compress multiple files in parallel.