zlib inflateReset causes memory leak (not)

Question

I am currently working on the below requirement. Here is the requirement: On the server side a large file is divided into 4000-byte blocks (frames). Each block is in turn compressed (using zlib) and sent to client process. For instance, if a file is 12000 bytes in size then it is divided into 3 blocks.

Above file will have 3 blocks => Block-0, Block-1, Block-2

On receipt, client decompresses each block (or frame) and writes to buffer allocated on the heap.When all the blocks corresponding to the entire file is received by the client, then the uncompressed version of the resultant file is written to the disk.

I have written a routine inflateData that does the following based on the block # received:

When the first block is received, - inflateInit - inflate - inflateReset When the intermediate blocks are received, - inflate - inflateReset When the last block is received, - inflate - inflateEnd

With the above routine, Decompression of blocks happens as expected. But the issue that I face is it consumes lots of memory and at some point entire system slows down. When checked with valgrind, memory leak is reported with inflateInit2_. This causes the system resources to be exhausted.

==30359== 57,312 bytes in 6 blocks are possibly lost in loss record 64 of 67

==30359== at 0x4A069EE: malloc (vg_replace_malloc.c:270)

==30359== by 0x3E57808F1E: inflateInit2_ (in /lib64/libz.so.1.2.3)

==30359== by 0x40C220: inflateData (productMaker.c:1668)

Below is the routine inflateData.

    int inflateData(
    char*    const    inBuf,                  
    unsigned long     inLen,                  
    unsigned int      isFirstBlk,
    unsigned int      isLastBlk,
    const    char*    outBuf,                 
    unsigned long*     outLen)                 
{

  int have;
  int readsz;
  int bsize;
  static z_stream zstrm;
  int zerr;
  int flush;
  char out[CHUNK_SZ];
  char in[CHUNK_SZ];
  int ret,nwrite,idx = -1;
  int savedByteCntr=0;
  unsigned char *dstBuf;
  int  firstCall = 1;
  int totalBytesIn=0;
  int inflatedBytes=0;
  int decompByteCounter = 0;
  int num=0;


  ret = Z_OK;
  readsz = 0;
  bsize = CHUNK_SZ;

  dstBuf = (unsigned char *) outBuf;

  if(isFirstBlk){
        memset(&zstrm, '\0', sizeof(z_stream));
        zstrm.zalloc = Z_NULL;
        zstrm.zfree = Z_NULL;
        zstrm.opaque = Z_NULL;

        if ((zerr = inflateInit(&zstrm)) != Z_OK) {
                uerror("ERROR %d inflateInit (%s)",
                        zerr, decode_zlib_err(zerr));
                return -1;
        }
  }

    while(totalBytesIn < inLen ) {
      int compChunkSize = ((inLen - totalBytesIn) > 5120) ? 5120 :
                                                  (inLen - totalBytesIn);
      memcpy(in, inBuf + totalBytesIn, compChunkSize);

      zstrm.avail_in = inLen - totalBytesIn;
      zstrm.next_in = in ;

      zstrm.avail_out = CHUNK_SZ;
      zstrm.next_out = out;
      inflatedBytes = 0;

      while(ret != Z_STREAM_END) {
         ret  = inflate(&zstrm, Z_NO_FLUSH);
         if(ret < 0) {
          uerror(" Error %d inflate (%s)", ret, decode_zlib_err(ret));
          (void)inflateEnd(&zstrm);
           return ret;
         }
         inflatedBytes = CHUNK_SZ - zstrm.avail_out;

         if(inflatedBytes == 0) {
              unotice("\n Unable to decompress data - truncated");
              break;
         }
     totalBytesIn += zstrm.total_in;
     decompByteCounter += inflatedBytes;

     memcpy(dstBuf + savedByteCntr, out, inflatedBytes);
     savedByteCntr = decompByteCounter;
   }

   // Reset inflater for additional input
   ret = inflateReset(&zstrm);
   if(ret == Z_STREAM_ERROR){
      uerror(" Error %d inflateReset (%s)", ret, decode_zlib_err(ret));
      (void)inflateEnd(&zstrm);
       return ret;
   }
  }

if(isLastBlk){
 ret = inflateEnd(&zstrm);
 if(ret < 0) {
   uerror("Fail inflateEnd %d [%s] ", ret, decode_zlib_err(ret));
   return (ret);
 }
}

  *outLen = decompByteCounter;
   return 0;
}

Thanks in advance for the support.

Thanks, Sathya.

Have you checked that `inflateEnd` is indeed called? We can't tell from the code snippet you have provided. — kaylum, Dec 23 '15 at 04:34

score 1 · Accepted Answer · answered Dec 23 '15 at 06:55

You are making an error in your use of your inflateData() routine.

First off, using a static variable in this way is a horrible idea. If you call your inflateData() twice with isFirstBlk true without an intermediate call with isLastBlk true, then you will wipe out the reference to the first set of allocations, resulting in a memory leak.

To avoid this sort of error, you should keep track of whether zstrm is initialized or not, and reject any attempt to initialize an already initialized stream. Better still would be to not even have an isFirstBlk, and simply initialize zstrm on the first call and on any call that immediately follows a call with isLastBlk true.

So you are either doing the above, calling twice with isFirstBlk true, or failing to call with isLastBlk true.

As per your advise, I have changed the code to not have static variable. Also initialize the stream when the first block is received and call inflateEnd after receiving the last block. Got rid of isFirstBlk and isLastBlk. Now valgrind does not report memory leak. Thanks for the help and support. — Sathya, Dec 24 '15 at 06:13

zlib inflateReset causes memory leak (not)

1 Answers1