How to get the uncompressed size of an LZMA2 file (.xz / liblzma)

Question

I'm looking for a way to get the uncompressed stream size of an LZMA2 / .xz file compressed with the xz utility.

I'm using liblzma from Windows/Linux for this task, so I guess I'm looking for some C/C++ API in liblzma that will do the trick.

score 6 · Accepted Answer · answered Feb 01 '10 at 02:32

I think I've found a solution.

This is a very crude code sample, but seems to work fine.

I'm assuming I have a do_mmap() function that maps the entire file as read-only into memory, and returns the total size mapped. This can naturally be adapted to use read/fread/ReadFile or any other File API.

extern size_t get_uncompressed_size(const char *filename)
{
   lzma_stream_flags stream_flags;
   int file_size;

   const uint8_t *data = (uint8_t *) do_mmap(filename, &file_size);

   // 12 is the size of the footer per the file-spec...
   const uint8_t *footer_ptr = data + file_size - 12;

   // Something is terribly wrong
   if (footer_ptr < data) {
     do_unmap((void *)data, file_size);
     return -1;
   }

   // Decode the footer, so we have the backward_size pointing to the index
   lzma_stream_footer_decode(&stream_flags, (const uint8_t *)footer_ptr);
   // This is the index pointer, where the size is ultimately stored...
   const uint8_t *index_ptr = footer_ptr - stream_flags.backward_size;
   // Allocate an index
   lzma_index *index = lzma_index_init(NULL);
   uint64_t memlimit;
   size_t in_pos = 0;
   // decode the index we calculated
   lzma_index_buffer_decode(&index, &memlimit, NULL, index_ptr, &in_pos, footer_ptr - index_ptr);
   // Just make sure the whole index was decoded, otherwise, we might be
   // dealing with something utterly corrupt
   if (in_pos != stream_flags.backward_size) {
     do_unmap((void *)data, file_size);
     lzma_index_end(index, NULL);
     return -1;
   }
   // Finally get the size
   lzma_vli uSize = lzma_index_uncompressed_size(index);
   lzma_index_end(index, NULL);
   return (size_t) uSize;
}

I want to use lzma2 native library to write hadoop compression codec.If you have any API documentation or your help in compression/decompression related understanding will be very helpful. — samarth, Apr 16 '12 at 12:47
Two issues. (1) You need to initialize `memlimit` before calling `lzma_index_buffer_decode`, or in some cases it will return `LZMA_MEMLIMIT_ERROR`. (2) You shouldn't call `lzma_index_init`. The initial value of `index` is ignored by `lzma_index_buffer_decode`. — Patrick, Oct 02 '13 at 16:42
Yes Patrick and replace 12 by LZMA_STREAM_HEADER_SIZE "Stream Header and Stream Footer have the same size" see lzma/stream_flags.h. — Stephane, Jun 19 '20 at 16:21

score 0 · Answer 2 · edited Nov 18 '22 at 12:24

Having downloaded the source from sourceforge and had a look here, I quoted this from the main header file LzmaLib.h

/*
LzmaUncompress
--------------
In:
  dest     - output data
  destLen  - output data size
  src      - input data
  srcLen   - input data size
Out:
  destLen  - processed output size
  srcLen   - processed input size
Returns:
  SZ_OK                - OK
  SZ_ERROR_DATA        - Data error
  SZ_ERROR_MEM         - Memory allocation arror
  SZ_ERROR_UNSUPPORTED - Unsupported properties
  SZ_ERROR_INPUT_EOF   - it needs more bytes in input buffer (src)
*/

MY_STDAPI LzmaUncompress(unsigned char *dest, size_t *destLen, const unsigned char *src, SizeT *srcLen,
  const unsigned char *props, size_t propsSize);

It looks that destLen is the size of the data that is uncompressed.

I think the XZ file format isn't straight LZMA, but rather a wrapper for a variety of compression algorithms — Dietrich Epp, Jan 31 '10 at 13:44

How to get the uncompressed size of an LZMA2 file (.xz / liblzma)

2 Answers2

Linked