13

I am using zlib to compress a stream of text data. The text data comes in chunks, and for each chunk, deflate() is called, with flush set to Z_NO_FLUSH. Once all chunks have been retrieved, deflate() is called with flush set to Z_FINISH.

Naturally, deflate() doesn't produce compressed output on each call. It internally accumulates data to achieve a high compression rate. And that's fine! Every time deflate() produces compressed output, that output is appended to a database field - a slow process.

However, once deflate() produces compressed data, that data may not fit into the provided output buffer, deflate_out. Therefore several calls to deflate() are required. And that is what I want to avoid:

Is there a way to make deflate_out always large enough so that deflate() can store all the compressed data in it, every times it decides to produce output?

Notes:

  • The total size of the uncompressed data is not known beforehand. As mentioned above, the uncompressed data comes in chunks, and the compressed data is appended to a database field, also in chunks.

  • In the include file zconf.h I have found the following comment. Is that perhaps what I am looking for? I.e. is (1 << (windowBits+2)) + (1 << (memLevel+9)) the maximum size in bytes of compressed data that deflate() may produce?

    /* The memory requirements for deflate are (in bytes):
            (1 << (windowBits+2)) +  (1 << (memLevel+9))
 that is: 128K for windowBits=15  +  128K for memLevel = 8  (default values)
 plus a few kilobytes for small objects. For example, if you want to reduce
 the default memory requirements from 256K to 128K, compile with
     make CFLAGS="-O -DMAX_WBITS=14 -DMAX_MEM_LEVEL=7"
 Of course this will generally degrade compression (there's no free lunch).

   The memory requirements for inflate are (in bytes) 1 << windowBits
 that is, 32K for windowBits=15 (default value) plus a few kilobytes
 for small objects.
*/
feklee
  • 7,555
  • 9
  • 54
  • 72
  • See http://stackoverflow.com/questions/4936255/zlib-how-to-dimension-avail-out – nos Jan 17 '12 at 23:17
  • 2
    @nos: This is only helpfull, if the size of the input is known. – Eugen Rieck Jan 17 '12 at 23:19
  • I read the comment in `zconf.h` to be the memory requirement for compression, not the size of the output buffer. That said, it seems logical, that an upper bound for the output buffer is the total memory requirements (128K+128K+"a few kilobytes" in the above example) + header length (40 byte). – Eugen Rieck Jan 17 '12 at 23:29

2 Answers2

7

deflateBound() is helpful only if you do all of the compression in a single step, or if you force deflate to compress all of the input data currently available to it and emit compressed data for all of that input. You would do that with a flush parameter such as Z_BLOCK, Z_PARTIAL_FLUSH, etc.

If you want to use Z_NO_FLUSH, then it becomes far more difficult as well as inefficient to attempt to predict the largest amount of output deflate() might emit on the next call. You don't know how much of the input was consumed at the time the last burst of compressed data was emitted, so you need to assume almost none of it, with the buffer size growing unnecessarily. However you attempt to estimate the maximum output, you will be doing a lot of unnecessary mallocs or reallocs for no good reason, which is inefficient.

There is no point to avoid calling deflate() for more output. If you simply loop on deflate() until it has no more output for you, then you can use a fixed output buffer malloced once. That is how the deflate() and inflate() interface was designed to be used. You can look at http://zlib.net/zlib_how.html for a well-documented example of how to use the interface.

By the way, there is a deflatePending() function in the latest version of zlib (1.2.6) that lets you know how much output deflate() has waiting to deliver.

Mark Adler
  • 101,978
  • 13
  • 118
  • 158
  • Thank you very much for this detailed answer! To predict the required output buffer for the next call to `deflate()`, I considered adding the size reported by `deflatePending()` and the value returned by `deflateBound()`. That's akin to the suggestion by @EugenRieck. However, as I understand it, this is not a good idea as `deflateBound()` is documented to work only when passed the size of the entire input to be compressed. I.e. `deflateBound()` is not documented to work for chunks of input. – feklee Jan 31 '12 at 21:56
  • 1
    deflateBound() could work for chunks of input, but only if all of the previous input had been compressed and emitted. This can only be assured by using a flush option other than Z_NO_FLUSH and consuming all of the output on the previous calls. In this case, deflatePending() would be useful for when Z_BLOCK or Z_PARTIAL_FLUSH is used, since they can leave a few bits behind. When using Z_NO_FLUSH, deflateBound() + deflatePending() would be missing the third piece, which is the size of the input consumed on the previous deflate() calls, but not yet compressed and emitted. – Mark Adler Feb 04 '12 at 20:12
2

While looking at the sources for a hint, I fell over 

/* =========================================================================
 * Flush as much pending output as possible. All deflate() output goes
 * through this function so some applications may wish to modify it
 * to avoid allocating a large strm->next_out buffer and copying into it.
 * (See also read_buf()).
 */
local void flush_pending(strm)
    z_streamp strm;
{
    unsigned len = strm->state->pending;
...

tracing the use of void flush_pending() throughout deflate() shows, that an upper bound on the needed output buffer in the middle of the stream is

strm->state->pending + deflateBound(strm, strm->avail_in)

the first part accounts for data still in the pipe from previous calls to deflate(), the second part accounts for the not-yet processed data of length avail_in.

Eugen Rieck
  • 64,175
  • 10
  • 70
  • 92
  • You were correct in the comment to my now-deleted answer. I was forgetting about the internal state. Out of curiosity, I looked at that pending value after the first call to deflate in a quick test. The avail_in was zero, avail_out was 2, and pending was zero (0). It did not seem to reflect the actual amount of pending data. The next call to deflate to flush it dumped ~8K to the output. So that may not be an accurate measurement... in at least one situation. – Mark Wilkins Jan 18 '12 at 00:54
  • You say that `strm->state->pending` is the size of *the data still in the pipe*. If I understand it correctly, then this size increases with every call to `deflate()`, until an unknown upper bound is reached. And this upper bound is precisely what I'm looking for. So how is this helpful? Am I missing something? – feklee Jan 18 '12 at 20:45
  • I meant, that if you give deflate() a buffer of size strm->state->pending + deflateBound(strm, strm->avail_in) it will never run out of buffer space. – Eugen Rieck Jan 18 '12 at 21:19
  • I see. So before calling `deflate()`, one would allocate `strm->state->pending + deflateBound(strm, strm->avail_in)` bytes of memory for `strm->next_out`. Thanks for figuring that out! Still I am not sure if I should rely on this method. After all it is not documented as part of the zlib API. – feklee Jan 30 '12 at 16:33
  • Documentation in `zlib.h` strongly suggests that accessing `strm->state` is a bad idea: `struct internal_state FAR *state; /* not visible by applications */` – feklee Jan 30 '12 at 16:59