Install data directory tree with massive number of files using automake

Question

I have a data directory which I would like automake to generate install and uninstall targets for. Essentially, I just want to copy this directory verbatim to the DATA directory, Normally, I might list all the files individually, like

dist_whatever_DATA=dir/subdir/filea ...

But the problem arises when my directory structure looks like this

*root
 *subdir
  *~10 files
 *subdir
  *~10 files
 *subdir
  *~700 files
 *subdir
 ...
 ~20 subdirs

I just cannot list all 1000+ files included as part of my Makefile.am. That would be ridiculous.

I need to preserve the directory structure as well. I should note that this data is not generated at all by the build process, and is actually largely short audio recordings. So it's not like I would want automake to "check" that every file I want to install has actually been created, as they're either there or not, and whatever file is there, I know I want it to be installed, and whatever file is not, should not be installed. I know that this is the justification used in other places to not do wildcard instsalls, but all the possible reasons don't apply here.

score 11 · Answer 1 · answered Jun 18 '11 at 15:53

11

I would use a script to generate a Makefile fragment that lists all the files:

echo 'subdir_files =' > subfiles.mk
find subdir -type f -print | sed 's/^/  /;$q;s/$/ \\/' >> subfiles.mk

and then include this subfiles.mk from your main Makefile.am:

include $(srcdir)/subfiles.mk
nobase_dist_pkgdata_DATA = $(subdir_files)

A second option is to EXTRA_DIST = subdir, and then to write custom install-data-local and uninstall-local rules.

The problem here is that EXTRA_DIST = subdir will distributes all files in subdir/, including backup files, configuration files (e.g. from your VCS), and other things you would not want to distribute.

Using a script as above let you filter the files you really want to distribute.

answered Jun 18 '11 at 15:53

adl

15,627
6
51
65

Just a note: The first one doesn't work, as it fails with "make[2]: execvp: /bin/bash: Argument list too long" – Jeremy Salwen Jun 21 '11 at 05:33
Wow! I am generating some `Makefile`s with pretty large lists of files, but had not *yet* realized I would hit the command line size limit some day! – adl Sep 09 '11 at 16:00
Could you give an example of the second option? If I just "cp -Rf" I get wrong permissions and I need to chmod after cp (or maybe umask before cp). I looked into automatically generated rules, they use $(INSTALL_DATA) instead of cp, but in this case all files need to be specified, so this would require "find", etc. What's the best, most portable way to install whole directory in install-data-local? – marcin Dec 14 '11 at 14:16
@marcin Something like this http://www.google.com/codesearch#EIS5BlthS9Y/Makefile.am&q=install-data-local%20INSTALL_DATA&type=cs – adl Dec 16 '11 at 07:37
1

The find command has a printf output option which might be easier to understand then piping to sed. – Ishmael Feb 14 '12 at 09:30
@Ishmael: the GNU implementation of `find` has a `-printf` option, but it this is not a standard option. If you want your Makefiles to be portable, you should avoid it. – adl Feb 14 '12 at 11:13

score 10 · Accepted Answer · answered Jun 20 '11 at 13:23

I've found that installing hundreds of files separately makes for a tormentingly long invocation of make install. I had a similar case where I wanted to install hundreds of files, preserving the directory structure, and I did not want to change my Makefile.am every time a file was added to or removed from the collection.

I included a LZMA archive of the files in my distribution, and made automake rules like so:

GIANTARCHIVE = My_big_archive.tar.lz

dist_pkgdata_DATA = $(GIANTARCHIVE)

install-data-hook:
    cd $(DESTDIR)$(pkgdatadir); \
    cat $(GIANTARCHIVE) | unlzma | $(TAR) --list > uninstall_manifest.txt; \
    cat $(GIANTARCHIVE) | unlzma | $(TAR) --no-same-owner --extract; \
    rm --force $(GIANTARCHIVE); \
    cat uninstall_manifest.txt | sed --expression='s/^\|$$/"/g' | xargs chmod a=rX,u+w

uninstall-local:
    cd $(DESTDIR)$(pkgdatadir); \
    cat uninstall_manifest.txt | sed --expression='s/ /\\ /g' | xargs rm --force; \
    rm --force uninstall_manifest.txt

This way, automake installs My_big_archive.tar.lz in the $(pkgdata) directory, and extracts it there, making a list of all the files that were in it, so it can uninstall them later. This also runs much faster than listing each file as an install target, even if you were to autogenerate that list.

Why do you install this into `$(pkgdatadir)`? Just extract it from `$(srcdir)` in `install-data-hook`. Use something like `EXTRA_DIST = giant_archive.tar.lzma` so it is distributed. — Jack Kelly, Jun 21 '11 at 06:01
Three minor suggestions, unrelated to the actual question. I would replace `cat xxx | yyy | zzz` by `yyyy — adl, Sep 09 '11 at 15:54
@adl, thanks! I always thought that using long options in scripts made the scripts easier to read, but portability is better. — ptomato, Sep 09 '11 at 16:50

score 3 · Answer 3 · answered Jun 19 '11 at 00:53

3

I would write a script (either as a separate shell script, or in the Makefile.am), that is run as part of the install-data-hook target.

answered Jun 19 '11 at 00:53

Jack Kelly

18,264
2
56
81

Install data directory tree with massive number of files using automake

3 Answers3

Linked