LZO compress char*

Question

I have installed LZO on my Ubuntu machine and I would like to use ti to compress a char* type string.

In the example files I have found this code snippet (I have already edited it just a little for my application):

  int r;
  lzo_bytep in;
  lzo_bytep out;
  lzo_voidp wrkmem;
  lzo_uint in_len;
  lzo_uint out_len;
  lzo_uint new_len;
  int strToCompressLen; // I have added this

  /*
   * Step 1: initialize the LZO library
   */
  if (lzo_init() != LZO_E_OK)
  {
    cout << "internal error - lzo_init() failed !!!" << endl;
    cout << "(this usually indicates a compiler bug - try recompiling\nwithout optimizations, and enable '-DLZO_DEBUG' for diagnostics)" << endl;
    //return 4;
  }

  // here I get the data I want to compress
  char* imageData = (char*)imageIn->getFrame();

  /*
   * Step 2: allocate blocks and the work-memory
   */
  strToCompressLen = strlen(imageData);
  in = (lzo_bytep) xmalloc(strToCompressLen);
  out = (lzo_bytep) xmalloc((strToCompressLen + strToCompressLen / 16 + 64 + 3));
  wrkmem = (lzo_voidp) xmalloc(LZO1X_1_MEM_COMPRESS);
  if (in == NULL || out == NULL || wrkmem == NULL)
  {
        cout << "out of memory" << endl;
        //return 3;
  }

  /*
   * Step 3: prepare the input block that will get compressed.
   *         We just fill it with zeros in this example program,
   *         but you would use your real-world data here.
   */
  in_len = strToCompressLen;
  lzo_memset(in,0,in_len);

  /*
   * Step 4: compress from 'in' to 'out' with LZO1X-1
   */
  r = lzo1x_1_compress(in,in_len,out,&out_len,wrkmem);
  if (r != LZO_E_OK)
  {
        /* this should NEVER happen */
        cout << "internal error - compression failed: " << r << endl;
        //return 2;
  }
  /* check for an incompressible block */
  if (out_len >= in_len)
  {
        cout << "This block contains incompressible data." << endl;
    //return 0;
  }

But what it does is it just fill in zeros. I need to compress a char* variable.

I guess I need to edit these lines:

  in_len = strToCompressLen;
  lzo_memset(in,0,in_len);

I have the string I want to compress in this variable:

  char* imageData = (char*)imageIn->getFrame();

Do I need to cast it to some other type?

The documentation to LZO is not very helpful (or maybe I just can't use it properly).

Answer 1

What type does getFrame() return? Why do you need to cast it to char*? The most obvious problem is using strlen() on binary data -- strlen stops when it encounters the first zero-byte.

Answer 2

There are three main points of confusion:

1. It's best not to use char* when the buffer the pointer is pointing to is not a null-terminated string as it's confusing.
2. strlen() will only give you the length of a null-terminated string, it won't give you the size of an arbitrary buffer in memory. You need to get that information elsewhere.
3. The buffer you pass to lzo1x_1_compress() actually needs to contain the data you want to compress and not an empty buffer full of zeros.

Assuming you can get the size of your image from imageIn using something like imageIn->getFrameSizeBytes(), try this:

  int r;
  lzo_bytep out;
  lzo_voidp wrkmem;
  lzo_uint out_len;
  lzo_uint new_len;

  /*
   * Step 1: initialize the LZO library
   */
  if (lzo_init() != LZO_E_OK)
  {
    cout << "internal error - lzo_init() failed !!!" << endl;
    cout << "(this usually indicates a compiler bug - try recompiling\nwithout optimizations, and enable '-DLZO_DEBUG' for diagnostics)" << endl;
    //return 4;
  }

  // here I get the data I want to compress
  lzo_bytep imageData = (lzo_bytep) imageIn->getFrame();
  size_t uncompressedImageSize = imageIn->getFrameSizeBytes();

  /*
   * Step 2: allocate blocks and the work-memory
   */
  out = (lzo_bytep) xmalloc((uncompressedImageSize + uncompressedImageSize / 16 + 64 + 3));
  wrkmem = (lzo_voidp) xmalloc(LZO1X_1_MEM_COMPRESS);
  if (out == NULL || wrkmem == NULL)
  {
        cout << "out of memory" << endl;
        //return 3;
  }

  /*
   * Step 4: compress from 'imageData' to 'out' with LZO1X-1
   */
  r = lzo1x_1_compress(imageData,uncompressedImageSize,out,&out_len,wrkmem);
  if (r != LZO_E_OK)
  {
        /* this should NEVER happen */
        cout << "internal error - compression failed: " << r << endl;
        //return 2;
  }

  /* check for an incompressible block */
  if (out_len >= uncompressedImageSize)
  {
        cout << "This block contains incompressible data." << endl;
    //return 0;
  }

Don't forget to free wrkmem. Even better, use C++ and std::vector for the working memory so it is freed automatically.