pulling file and zip while pulling in UNIX

Question

I am generating files from netezza server using nzsql. Now the next part of the requirement is to zip and move the file into a different box (a second unix box). I can achieve this sitting in the 1st box (sitting in the first box I am first doing gzip and then moving the file using scp). But my question is, can this be done directly from the 2nd box. Means in the 1st box there will be the files which will be generated nzsql and in the 2nd box that file will be pulled and while pulling it will be zipped (using mkfifo or by any other means). In the 1st box I don't want the zip file to be created or resides. Please let me know how this can be achieved.

Answer 1

you can in your box 2, have this running periodically:

rsync -azP user@box1:/path/to/your/sync/folder /dst/folder

the parameter explanation are directly copied from man page:

-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)

-r, --recursive recurse into directories

-l, --links copy symlinks as symlinks

-p, --perms preserve permissions

-D same as --devices --specials

--devices preserve device files (super-user only)

--specials preserve special files

-t, --times preserve modification times

-o, --owner preserve owner (super-user only)

-g, --group preserve group

-z, --compress compress file data during the transfer

-P same as --partial --progress

--partial keep partially transferred files

but i doubt whether it's proper or not. you should push it from box1 to minimize the bandwidth usage. pulling is almost the same as polling.

reference here is a very complete tutorial: https://www.digitalocean.com/community/tutorials/how-to-use-rsync-to-sync-local-and-remote-directories-on-a-vps