1

I have a large directory of files and I would like to maintain a single compressed tar archive that I can server over http containing all the files which is brought up to date once each day.

Many of the files will not change day-to-day and I'd like to avoid hours of processor time compressing the same files every day.

tar "cannot update compressed archives", so tar uj won't help.

Is there a clever way to do this?

artfulrobot
  • 2,949
  • 13
  • 36
  • 60
  • Related question http://superuser.com/questions/774777/appendable-and-greppable-archive-on-centos-to-hold-many-log-files-logrotate-on – kubanczyk Jun 29 '14 at 15:32

2 Answers2

0

Because this is accessed over http, use PHP to generate the tar file on the fly using something like this:

<?php
    set_time_limit(1);
    header("Pragma: public"); // required
    header("Expires: 0");
    header("Cache-Control: must-revalidate, post-check=0, pre-check=0");
    header("Cache-Control: private",false); // required for certain browsers
    header("Content-Type: application/x-bzip2; charset=binary");
    header("Content-Disposition: attachment; filename=\"archive.tar.bz2\";" );
    passthru("tar cj --exclude-vcs /path/to/files",$err);
    if ($err) {
        error_log("exit value: $err");
    }
    exit;

Obviously this does not help with the case that this file is going to be downloaded a lot.

artfulrobot
  • 2,949
  • 13
  • 36
  • 60
0

This idea was on a competitor advice forum.

tar cf --newer YYYYMMDD

    To copy differences or only the files since the last tar date 
    here is the command.

    Let's say we did a tar on Feb 9th, 2009 as follows:
    (cd /mydata; tar cf - *) | tar xvf -

    Today is Feb 11th and we only want to copy the files that have changed 
    since Feb 9th, 2009. The command would be
    (cd /mydata; tar cf - --newer 20090209 * ) | tar xvf -
ndasusers
  • 427
  • 1
  • 5
  • 14