8

I want my program to be able to use zlib, lzma, lzo and bzip2 compression algorithms.

Is there any compression library for C which simlifies working with multiple algorithms (like libmcrypt supporting multiple encryption modes and algorithms)?

Expecting something like this:

struct compressor c;
universal_compressor_init(&c, "lzma", 7 /* compression level */);
universal_compressor_compress(&c, inputbuf, inputsize, outputbuf, &outputsize);
universal_compressor_save_state(&c, statebuf, &statesize);

Note: It is not about zip/rar/7z/tar/cpio and other archive formats, it's about compression of raw buffers. Think of compressed networking protocol or about random access to compressed block device (like cloop).

Vi.
  • 37,014
  • 18
  • 93
  • 148

2 Answers2

3

LibArchive fulfills your requirements.

From the introduction:

The libarchive library features:

  • Support for a variety of archive and compression formats.
  • Robust automatic format detection, including archive/compression combinations such as tar.gz. ...

EDIT: for handling raw streams (at least until they split libfilter from libarchive), consider using Boost::iostreams or libbu++ (just found on GitHub)

Stefano Sanfilippo
  • 32,265
  • 7
  • 79
  • 80
  • Is it suitable for many little independent headerless compressed buffers? – Vi. May 04 '13 at 22:26
  • Looks like primary goal of libarchive is working with archives as bundles of files, not as compression embedded in some protocol. – Vi. May 04 '13 at 22:29
  • Can I use libarchive filters (without the rest libarchive) with libarchive's public API? – Vi. May 04 '13 at 23:45
  • Unfortunately, not: that feature is planned for a future release. Updated answer. – Stefano Sanfilippo May 05 '13 at 00:25
  • If I understood what you are about to do, you might find squashfs very interesting with respect to block compression. – Stefano Sanfilippo May 05 '13 at 00:37
  • I also want snapshotting (means efficient storage of similar versions of some big file)... It should be simple and support large volumes of data efficiently. – Vi. May 05 '13 at 00:46
  • Then you could try to implement snapshotting over squashfs by taking advantage of the "append" mode or aggregating it with aufs &co.. Try playing around with [mksquashfs](http://manpages.ubuntu.com/manpages/hardy/man1/mksquashfs.1.html) and simmetrical companion unsquashfs. Caveat: it is a readonly FS. If you still need to do raw stream compression/decompression programmatically, then I suggest you to use Boost. – Stefano Sanfilippo May 05 '13 at 10:59
  • let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/29433/discussion-between-vi-and-esseks) – Vi. May 05 '13 at 11:00
0

I recommend 7-zip (http://www.7-zip.org/sdk.html). I haven't found anything to beat it for both compression speed and size.

Scott Jones
  • 2,880
  • 13
  • 19
  • I want to compress big file preserving random access inside it. So I want to compress little blocks and store index. Does 7-zip provide an API to do it efficiently? – Vi. May 04 '13 at 22:27
  • Yes, that was one of the needs I had (fast single file extraction from large archives). You might post on the 7-zip forum for more info: http://sourceforge.net/p/sevenzip/discussion/45797 – Scott Jones May 04 '13 at 22:32
  • Will it really handle 13107200 little files of 4096 bytes each well? – Vi. May 04 '13 at 22:34
  • Good question - I'm sure it would only take a few lines of code to find out. – Scott Jones May 04 '13 at 22:36
  • 7-zip's primary algorithm (lzma) is in fact listed in question. The purpose of the library in question is to be able to switch algorithms easily. – Vi. May 04 '13 at 22:43