2

I would like to use pigz to compress massive tar archives.

I am using cygwin. Unfortunately, pigz is not one of the standard cygwin packages.

Anyone know how to install pigz under cygwin?

Below are the 2 techniques I tried without success:

1) The README on this webpage (or in the README file, if you download the source from here) says that you should be able to build it from source merely by

Type "make" in this directory to build the "pigz" executable.

When I do that on my machine, I get a ton of warnings starting with

pigz.c:2950:20: warning: unknown conversion type character 'j' in format [-Wformat=]
                        (intmax_t)g.in_tot, (intmax_t)len, tag);

and then this final error:

gcc  -o pigz pigz.o yarn.o try.o deflate.o blocksplitter.o tree.o lz77.o cache.o hash.o util.o squeeze.o katajainen.o -lm -lpthread -lz
pigz.o:pigz.c:(.text+0xd4f8): undefined reference to `fsync'
collect2.exe: error: ld returned 1 exit status
make: *** [pigz] Error 1

That about exhausts my ability to build programs from source...

2) It looks like there is an old 2015 port of pigz version 2.3.3 to Cygwin Ports, the expanded cygwin package repository.

But that version out of date (the latest pigz is 2.4). Indeed, it looks like Cygwin Ports has migrated to github and searching there for pigz there finds nothing.

I am not even sure how to use Cygwin Ports! The project's homepage merely says

Follow the normal Cygwin installation instructions in order to install any of the packages currently maintained by this project.

I assume that that means to run cygwin's setup-x86.exe, but when it asks you to "Choose A Download Site" you will need to enter some URL for Cygwin Ports.

Web searching found little information. This link says to use http://sourceware.org/cygwinports/ but setup-x86.exe soon generated an error for that URL. The instructions in this link also did not work for me.

HaroldFinch
  • 762
  • 1
  • 6
  • 17
  • It built fine for me from source. What version of Cygwin and GCC are you using? – varro Jan 26 '18 at 23:08
  • same here. It builds with no warnings at all – matzeri Jan 27 '18 at 05:23
  • @varro and matzeri: my initial post was done on my work computer. I am home now. On my home machine, I too eventually got pigz to compile perfectly. Altho, I first had to run cygwin's setup-x86_64.exe three times to install: make, 3 gcc C/C++ related packages, 3 zlib related packages. For each of those installs, I also installed all the dependencies that setup-x86_64.exe suggested. – HaroldFinch Jan 28 '18 at 04:16
  • @varro and matzeri: My work computer differs in several ways from my home computer. Now that I think about it, the worst way is that it has [RTools](https://cran.r-project.org/bin/windows/Rtools/) installed. RTools is awful in many ways. Almost certainly relevant here, it has some old but specific 32 bit versions of certain cygwin commands that it requires. As a consequence, RTools has to be first on my Windows Path env var, and my cygwin install must also be 32 bit; see [this link](http://r.789695.n4.nabble.com/RTools-and-previous-Cygwin-installation-conflict-td4686674.html). – HaroldFinch Jan 28 '18 at 04:17
  • @varro and matzeri: I bet that the old cygwin commands in the RTools directory are screwing me up. When I get back to work, I will try temporarily removing RTools from my Path and see if it then builds. – HaroldFinch Jan 28 '18 at 04:18

3 Answers3

1

The C99 standard specifies the j specifier for printf(). (Note that the 99 refers to 1999. It is now 2018.) You can force the pigz compilation to not assume C99 by changing __STDC_VERSION__-0 >= 199901L || __GNUC__-0 >= 3 to 0. Then it won't try to use j.

Please let me know what the values of __STDC_VERSION__, __GNUC__, and __GNUC_MINOR__ are for your compiler.

Also pigz requires POSIX compliance, which would provide the fsync() call. You can just delete the reference to fsync(), which would just result in the --synchronous and -Y options having no effect.

Mark Adler
  • 101,978
  • 13
  • 118
  • 158
  • Considering that this program builds cleanly for other people without having to fiddle with the source, it would be better to figure out what's wrong with the OP's environment that do source changes. – varro Jan 27 '18 at 18:52
  • @Mark: thanks for your reply--I am honored to hear back from pigz's creator! Any chance that pigz can be added to the cygwin package list? – HaroldFinch Jan 28 '18 at 04:22
  • I would have no idea who does that or how. – Mark Adler Jan 28 '18 at 04:24
  • @Mark: I just benchmarked tar only, tar --> standard gzip, tar --> pigz (using all my cores), and tar --> xz (using all my cores). I archived both to local as well as NAS drives. I did this on an old Dell Precision 7500 workstation (~7 years old, albeit with 2 Xeon 5680s, so 12 cores total). Main result: pigz is ~10X faster than gzip. That is awesome! You have got to get pigz added as standard to all distros! In the case of cygwin, I am sure that the place to start asking is their main cygwin mailing list mentioned [here](https://cygwin.com/lists.html). – HaroldFinch Jan 29 '18 at 16:38
  • Great that the author of pigz has answered! +1 just for that. – Binarus Mar 12 '21 at 11:18
0

To follow up on comments above that I had with @varro and matzeri, I can now answer my own question: my suspicion was correct: RTools was the culprit. I found that if I temporarily removed all RTools elements from my Windows Path env var (for me: c:\Rtools\bin and c:\Rtools\mingw_32\bin), then I was able to get pigz make to work.

After doing this Path edit, I uninstalled my existing cygwin, reinstalled cygwin, installed my usual extra packages (chere, openssh, subversion, zip, unzip) and all their dependencies, installed make and all its dependencies, installed gcc-core (is the C compiler) and all its dependencies. At that point, I was able to make pigz perfectly.

HaroldFinch
  • 762
  • 1
  • 6
  • 17
0

There is a much easier way than compiling yourself. I had the same problem, and with a little bit of research found multiple ready-made .exe files (pigz.exe) for direct usage in Windows. I am using this one:

https://sourceforge.net/projects/pigz-for-windows/files/

The OP's main concern was: "I would like to use pigz to compress massive tar archives.", and I hope that this is a useful answer to that concern, although it does not explain how to get around the compiling problems.

Some additional notes:

The interesting thing that some folks may not be aware of is that nothing keeps us from using normal Windows binaries from within Cygwin, and vice versa. That is, even if the OP had sophisticated Cygwin / bash (or whatever) scripts which drive pigz and the whole process of compressing, he could use the ready-made pigz native Windows version linked above.

With or without Cygwin, there is no need to compile pigz yourself, unless you want the latest features or bug fixes.

Personally, I am using the native Windows pigz version from within Cygwin since a while. AFAIK, pigz has no progress bar, which is somehow inconvenient for me (from time to time I have to compress a single huge file (around 60 GB)). A convenient way to get around this is the pv utility. Since I haven't found a native Windows version of it, and since I am too lazy to compile it for Windows myself, I am using Cygwin's pv to display the progress when I let the native Windows pigz compress those huge files.

Binarus
  • 4,005
  • 3
  • 25
  • 41