-1

I'm trying to get rsync just to sync files that have changed and or been created since it's last run. This is an attempt to stop my NAS servers disks from spinning up every time rsync runs even when there is no file to copy across.

I'm assuming the easiest way to set the time of the last run is to touch a markerfile. That I can do. Where I am stuck is getting rsync to only sync files newer than the markerfile. I don't think rsync has a -newer option. I'm assuming I might need to run some other commands that will create an include list but I am a real *nix noob.

If it matters, the scripts will be running on the unRAID version of slackware.

dalben
  • 21
  • 1
  • 1

4 Answers4

3

Your questions seems flawed by this part:

"This is an attempt to stop my NAS servers disks from spinning up every time rsync runs even when there is no file to copy across."

If the disks don't spin up how are you going to check for newer versions of the files stored on them? rsync will only sync files that are newer or have been changed on the source (unless using non-default option like -c, --checksum skip based on checksum, not mod-time & size.

If you don't want to spin up the disks you can, as you say use a marker file on another disk that is always spinning I suppose. So you could write a simple bash script that fudges this together but is sounds like the wrong approach to me.

jwbensley
  • 4,202
  • 11
  • 58
  • 90
  • The files are sitting on a drive (ssd) outside the raid that is always spun up. Currently Rsync needs to spin up the raid drives to see what's there to determine what to copy. I'm trying to avoid the spin up of drives when no copy is needed. – dalben Mar 29 '13 at 16:53
  • @dalben That doesn't make any sense; the files our outside the raid on another drive that is always running, so why does the RAID need to spin up if there files aren't there? You're not being very clear – jwbensley Mar 30 '13 at 15:34
  • RSYNC checks the destination to see if the source files have changed (new files or modified). That check spins up the RAID disks. – dalben Apr 01 '13 at 11:57
  • RSYNC checks the destination to see if the source files have changed (new files or modified). That check spins up the RAID disks. When I rsync new/modified files across I keep them on the source as well. I'm trying to stop the unnecessary raid spin up by adding some smarts to the script so that rsync only looks to run when there have been changes to the source since the last time it ran. I'm not sure what's so flawed about the question or the explanation. SvenW below seems to have understood it perfectly. – dalben Apr 01 '13 at 11:57
  • Well you have just written "RSYNC checks the destination to see if the source files have changed" - That doesn't make sense at all! Rsync will check the source to see if the source files are newer by comparing their dates with the destination files at the specified destination. My point above is that if the source files are on an RAID, you have to spin up the array to do anything with them, including check their last modified date. You can use a marker file outside of the array (for example) but then you are syncing only if the marker file exists (for example) not if the files have changed. – jwbensley Apr 02 '13 at 13:50
  • Do you in fact mean that the RAID is the rsync destination, and you want to avoid spinning up the destination RAID? – jwbensley Apr 02 '13 at 13:50
  • >> Do you in fact mean that the RAID is the rsync destination, and you want to avoid spinning up the destination RAID? Yes. Was it really that hard to figure out ? – dalben Apr 10 '13 at 13:48
  • I shouldn't have to figure it out, is it that hard to write? – jwbensley Apr 10 '13 at 14:16
1

Your solution is to start rsync only if it needs to run. Write a script that checks if there are newer files on the source and if there are, start rsync.

A super-simple cron-ready sample script

Assuming a cron job running every 60 minutes with both source and target locally mounted, adapt accordingly (including the path to find and rsync (use which find and which rsync to get this): 

#!/bin/bash
FIND=/usr/bin/find
RSYNC=/usr/bin/rsync
SOURCE=/sourcedir
TARGET=/targetdir
INTERVAL=60
FILELIST=/tmp/filelist.$(date +%s)


cd $SOURCE
$FIND . -type f -mmin -$INTERVAL > $FILELIST
cat $FILELIST

if [ $(cat $FILELIST | wc -l) -ne "0" ]
then
   $RSYNC -av --files-from=$FILELIST $SOURCE/ $TARGET/
fi
rm -f $FILELIST
Sven
  • 98,649
  • 14
  • 180
  • 226
  • Yeah, that might make more sense than trying to get Rsync to determine what needs to be copied. Any idea what sort of commands could do this ? I'm seriously a noob but I can hack away at someone else's code when I have too – dalben Mar 29 '13 at 16:57
  • See my edit for a sample. – Sven Mar 29 '13 at 17:38
  • Thanks. I'll have a play and see what I can come up with. – dalben Mar 30 '13 at 03:29
0

try using rsync with the -u option, only sync's files which have been modified. Still causes disk to spin however reduces that actual execution time.

Michael Hodgdon
  • 1
0

You should use lsyncd or even inotifywait + scp instead, but it's a bit more scripting.

lsyncd

lsyncd uses inotify to detect if there's any new/modified files then use rsync+ssh do transfer the files.

inotifywait

If lsyncd is too complicated, you can use inotifywait, e.g.:

while read line
do
    echo "$line"
done < <(inotifywait -mr "/dir/to/monitor/")

(you'd need to change the echo part to scp)

rsync

If you use rsync, you can limit which files rsync should inspect using 'find' command, e.g.:

cd /source/dir/
rsync -ravhz --files-from=<(find . -mtime -1) ./ /target/dir/

The above rsync should not inspect files with modified/created time older than a day.

DISCLAIMER: I have not tested the above! Please be careful!!!

Community
  • 1
onlinesid
  • 1
  • 2