0

I try to use libarchive to create some archives. I use directly an example from wiki. Problem is that it stores everything in memory until archive_write_free is called. Simple measurement: adding about 2000000 x 1.5KB of random ascii buffers (as 7zip) stores ~350MB in memory, without single write into disc. I tried archive_write_open callback-version and write callback was never called until archive_write_free. Do you know any way to force libarchive to flush data in progress? or set some memory limit? or how to measure in runtime how much it allocated (I could close it, rotate file, and open new one)?

PS. I tried libarchive 3.2.2 from official Ubuntu 18.4 bundle and 3.6.1 from repo, both the same results.

Bogdan
  • 984
  • 8
  • 16
  • I really have no idea, but perhaps `archive_write_finish_entry` before (or after) the `archive_entry_free` call? – Some programmer dude Aug 09 '22 at 12:24
  • Why is it a problem that things are kept in memory for a while? – Jesper Juhl Aug 09 '22 at 12:25
  • @JesperJuhl problem is that _all_ things are kept in memory for whole processing, and it works on resource-limited machine – Bogdan Aug 09 '22 at 12:31
  • 1
    If it doesn't seem possible, perhaps add it as an [issue](https://github.com/libarchive/libarchive/issues) (if it doesn't exist already)? – Some programmer dude Aug 09 '22 at 12:34
  • @Someprogrammerdude `archive_write_finish_entry` doesn't help. I will report an issue. – Bogdan Aug 09 '22 at 12:37
  • @Bogdan I would expect the operating system to page out unused (or rarely used) data to disk if resources are limited. Why would your (user space) application need to care explicitly? – Jesper Juhl Aug 09 '22 at 12:49
  • @JesperJuhl I cannot allow the system to go outside its limits and on swap. I would accept lower compression level because of flushing data in the middle instead of pushing the system on swap because the lib consumed too much. – Bogdan Aug 09 '22 at 13:31

1 Answers1

0

It seems that it is a bug/feature of archive_write_set_format_7zip. When gzip or xz filter used then data are dumped frequently into disc as expected. I have reported an issue, maybe I will get more info there.

Bogdan
  • 984
  • 8
  • 16