3

I'm trying to read a compressed binary .x mesh file but my decompression is failing. The file is basically some directx header info and then a bunch of data in MSZIP format (i.e. 2 bytes are an int blockSize, 2 bytes are a "magic number" and then there are blockSize deflated bytes and then repeat until there's no more data) so for each block I'm just getting the compressed bytes and deflating like so-

internal static byte[] DecompressBlock(byte[] data) {
    var result = new List<byte>();
    var ms = new MemoryStream(data);
    var ds = new DeflateStream(ms, CompressionMode.Decompress);
    var newStream = new MemoryStream();
    ds.CopyTo(newStream);
    ds.Flush();
    ds.Close();
    return newStream.GetBuffer();
}

The first block deflates as expected. Subsequent blocks are the right inflated size but, seemingly randomly, some bytes are 0 when they shouldn't be, usually in in groups of 4-12.

How I can deflate different blocks of zipped data while maintaining the same history buffer?

UPDATE: After a little more research it looks like in MSZIP compression these blocks ARE the results of separate deflate operations but the "history buffer" is maintained between them I don't know if deflatestream is gonna be able to handle this. updated the actual question

Ross Manson
  • 33
  • 4

2 Answers2

4

Yes, there is something that you are missing. Each deflate block can and does use history from the previous deflate block. So at each block you must initialize the deflate dictionary with the last 32K of uncompressed data from the previous blocks.

Mark Adler
  • 101,978
  • 13
  • 118
  • 158
  • Thanks! I didn't see this before I updated the question. I don't suppose you'd know of a way to do this in c# would you? – Ross Manson Sep 08 '16 at 14:12
  • It does not appear that the Deflate class provides access to zlib's `deflateSetDictionary()` function. You would need to use zlib directly. – Mark Adler Sep 08 '16 at 15:25
  • Actually, if you're using zlib directly, you can use the undocumented `deflateResetKeep()` function that I added just for this purpose. It starts a new deflate stream, but keeps the dictionary from the deflate just completed using that state. There is also a corresponding `inflateResetKeep()`. – Mark Adler Sep 08 '16 at 16:19
  • Thanks! nEeD MorE chArACteRs tO coMMent – Ross Manson Sep 08 '16 at 16:20
0

If anyone is trying to read/write the MSZIP format for DirectX meshes using C# only, I found out it is possible to do it with SharpZipLib.

For reference, the format of the compressed DirectX file is the following:

  1. 16 bytes -> DirectX header
  2. 4 bytes -> total uncompressed file size (including the 16 bytes of its header)
  3. 2 bytes -> size of uncompressed block, up to 32KB
  4. 2 bytes -> size of compressed block, plus 2 (because it includes the magic number's size)
  5. 2 bytes -> magic number ("CK" in ascii)
  6. a compressed block

Parts 3-6 are repeated until the end of file. In practice all blocks but the last will be 32KB when uncompressed.

To decompress the file, you will have to implement the logic that extracts each compressed block, then give them to SharpZipLib's Zip.Compression.Inflater class. Reuse the same inflater for all blocks, but call its Reset method between each block.

This partly works, but you may get the "broken uncompressed block" error for some blocks. To overcome this problem, you will have to modify SharpZipLib's source, specifically the file Inflater.cs. The change is trivial - all you have to do is to disable/skip the DECODE_STORED_LEN2 case and go directly to DECODE_STORED instead.

To compress a file, split its contents into 32KB blocks, and the same logic applies: feed each block to the same Deflater and call Reset between each call. Again, you will have to modify the file DeflaterHuffman.cs, removing the line pending.WriteShort(~storedLength).

panoskj
  • 93
  • 5