How to syncronize XML files and store them as .gz at destination using rsync

Question

I'm using rsync -z to backup a large amount of XML files. The source is a huge machine with thousands of GBs available on disk, but the destination is not that powerful.

What I want to do is to compress XML files during transfer and keep them compressed at the destination. The option -z does the compression only during the transit.

Is it possible to do it using just one crontab command?

This is the command I am currently using:

*/30 * * * * rsync -rz -e 'sshpass -p "MY_PASSWORD" ssh -p 5022' --ignore-existing --quiet --delete REMOTE_USER@REMOTE_HOST:/path/to/remote/ /path/to/local/

Why aren't they gzipped on the source? If you don't gzip them at the source then rsync will try to transfer them again and again each time you run it, which will waste even more bandwidth and time. — Michael Hampton, Jul 22 '20 at 19:54
The permission of REMOTE_USER is read only. And the REMOTE_HOST belongs to a large company that has very strict security policies — optimus_prime, Jul 22 '20 at 19:58

score 0 · Answer 1 · answered Jul 23 '20 at 12:26

I managed to solve this with a workaround. On local host, I just compress the syncronized file and overwrite the original file with the string "0". It prevents rsync from synchronizing the same file again and significantly reduces disk load.

exec('find /path/to/local/ -name "*.xml" -size +10k',$output);
foreach ($output as $key => $filepath) {`
        
        //gzip the xml file
        $gz = gzopen("/path/to/gz/".basename($filepath).".gz", "wb9");
        gzwrite($gz, file_get_contents($filepath));
        gzclose($gz);
        
        //overwrite xml file
        $xml = fopen($filepath,"w");
        fwrite($xml, "0");
        fclose($xml);
}

How to syncronize XML files and store them as .gz at destination using rsync

1 Answers1