PHP LFTP data mirror output

Question

I'm using a Linux local computer and need to backup/mirror some very large file structures regularly. I only have access to SFTP.

I was after a simple one click solution. I originally tried to write the little script in BASH but I've never used it before and am not up to scratch with the syntax so I resorted to PHP. (I do understand PHP is not designed for this kind of work, but I'm on a tight time scale and don't have the time to get into BASH atm)

<?php
//init
parse_str(implode('&', array_slice($argv, 1)), $_GET);
$error = array();

$lPrefix         = '/home/hozza/Sites/';
$archiveLocation = '/home/hozza/Backups/';

$lDir = isset($_GET['l']) ? $_GET['l'] : $error[] = 'Local Directory Required';
$rDir = isset($_GET['r']) ? $_GET['r'] : $error[] = 'Remote Directory Required';
$bookmark = isset($_GET['b']) ? $_GET['b'] : $error[] = 'lftp Bookmark Required';

//Check for args
if(count($error) == 0) {
    $archiveName = end(explode('/', $lDir)) . '_' . date('Y-m-d_H-i');

    //Validate local dir
    if(is_dir($lPrefix . $lDir)) {

        //preserve Sublime Text 2 config SFTP files
        $ST2_SFTP_conf = false;
        if(file_exists($lPrefix . $lDir . '/sftp-config.json')) {
            $ST2_SFTP_conf = file_get_contents($lPrefix . $lDir . '/sftp-config.json');
            unlink($lPrefix . $lDir . '/sftp-config.json');
        }

        //Start mirror
        $lftOutput = explode("\n", shell_exec('lftp -e "mirror -e -p --parallel=10 --log=' . $archiveLocation . 'logs/' . $archiveName . '.txt ' . $rDir . '/ ' . $lPrefix . $lDir . '/; exit top" ' . $bookmark));

        //Tar regardless of lftp error or success
        $tarOutput = shell_exec('cd ' . $lPrefix . ' && tar -czf ' . $archiveLocation . $archiveName . '.tar.gz ' . $lDir);

        //Output completion or errors
        shell_exec('notify-send -i gnome-network-properties -t 0 "Mirror & Archive Complete" "' . $archiveName . '\n\n' . implode('\n', $lftOutput) . $tarOutput . '"');

        //put back ST2 SFTP conf
        if($ST2_SFTP_conf != false) file_put_contents($lPrefix . $lDir . '/sftp-config.json', $ST2_SFTP_conf);

        exit;
    }
    else shell_exec('notify-send -i error -t 0 "Mirror & Archive Error" "' . date('Y-m-d') . ' ' . date('H-i') . '\n' . $lDir . ' \n Does not exist! D:"');
}
else shell_exec('notify-send -i error -t 0 "Mirror & Archive Error" "' . date('Y-m-d') . ' ' . date('H-i') . '\n' . implode('\n', $error) . '"');
?>

It can be run for many sites via a short-cut like so...

terminator -T "Mirror & Archive" -e "php ~/Programs/mirror.php l=local-dir_path r=./ b=lftp-bookmark-name"

If no password is in the LFTP bookmark (there shouldn’t be as it's stored in plain text) the terminal prompts for a password, after the script has run, a nice notification is given with some info about files/folders/speed etc.

However, when the script is running in a terminal, only the "input password" bit is output to the terminal, I would like all the output displayed in the terminal (normally that would display what file/folder is currently working with etc.)

Anyone know how to do that?

Maybe you should catch/output the returns from your [`shell_exec()`](http://ca1.php.net/manual/en/function.shell-exec.php) calls? However that will only show the output *after* the command has fully completed. The way you are writing your script I don't think you'll ever be able to see it live. — Sammitch, Jan 07 '13 at 16:16
Yes atm it outputs the end result, I though it might not be possible like this. Do you have any advice on how to code it better for this situation? — hozza, Jan 07 '13 at 16:28

score 3 · Answer 1 · answered Jan 07 '13 at 18:14

IIRC the reason that you see the password prompt output to the terminal is that it is using stderr. You could try redirecting stdout to stderr for your commands which should show you the 'real-time' progress. Tack this on to the end of the shell_exec() command: 1>&2

ie:

shell_exec('lftp -e "mirror -e -p --parallel=10 --log=' . $archiveLocation . 'logs/' . $archiveName . '.txt ' . $rDir . '/ ' . $lPrefix . $lDir . '/; exit top" ' . $bookmark . ' 1>&2')

However, this will preclude you from having anything returned by shell_exec for logging purposes. What I would suggest is something like:

$log_stem = '/tmp/' . time() . '_';  // ie: /tmp/1357581737_
$lfOutfile = $log_stem . 'lftp.log';
$tarOutfile = $log_stem . 'tar.log';

shell_exec('lftp -blah | tee ' . $lfOutfile ' 1>&2' );
shell_exec('tar -blah | tee ' . $tarOutfile ' 1>&2' );

$lfOut = file_get_contents($lfOutfile);
$tarOut = file_get_contetns(tarOutfile);

// remove tmp files
unlink($lfOutfile);
unlink($tarOutfile);

Which will capture a copy of the output to a file before redirecting the output to stderr so you can watch it live.

However, if you want to run this via cron I would recommend against writing anything to stderr that is not an error, otherwise cron will send a warning email every time it is run.

I tried that but it does not work... The script does it's job just fine but it is still not output to the terminal. http://pastebin.com/zCeNvK1s — hozza, Feb 10 '13 at 14:31

score 0 · Answer 2 · answered Mar 10 '13 at 22:23

I think the last answer was close:

Either this:

shell_exec('lftp -blah |& tee ' . $lfOutfile );
shell_exec('tar -blah |& tee ' . $tarOutfile );

Or if that still doesn't work try this:

shell_exec('lftp -blah 2>&1 | tee ' . $lfOutfile );
shell_exec('tar -blah 2>&1 | tee ' . $tarOutfile );

PHP LFTP data mirror output

2 Answers2