4

I have a file storage server which stores files on disk using the file's sha256 hash as the filename, along with the file extension, and in three levels of directories, e.g. a PDF file with sha256 hash AABB1F1C6FC86DB2DCA6FB0167DE8CF7288798271EA24B68D857CBC5CF8DC66A would be stored in a subdirectory like this:

<root>/AA/BB/AABB1F1C6FC86DB2DCA6FB0167DE8CF7288798271EA24B68D857CBC5CF8DC66A.pdf

Files will be added to the directory structure, but never be deleted or modified.

I keep a live copy of this file-structure using a cron job running every 10 mins that uses rsync to push the files to a remote server. Since files are never deleted or changed once added, in practice it only sends new files.

I've found that the bandwidth used by rsync just for comparing the two directories (i.e. there were no changes) is about 11 MB and increasing as the total number of files grows (148 207 at the moment). It makes sense - rsync would in effect have to send a list of all of the filenames to the remote server to figure out which are missing on the remote server.

So my question is: is there a way to reduce the bandwidth used? It doesn't have to be an rsync-based solution, but it would be preferable. I was thinking of limiting the files that rsync looks at to only recently modified files, i.e. modified after the last sync, but it seems that is not recommended: rsync only files created or modified after a date and time

Any other suggestions?

Community
  • 1
Barry Pitman
  • 165
  • 5

3 Answers3

3

It's unrecommended for most cases, but given that your goal is to reduce the difference calculation bandwidth, it's appropriate. Consider the following script flow:

  1. touch a file to be your "high bar", this needs to be systematically named and not overwrite your last "high bar", which is now your "low bar". The script will transfer anything with mtime between those two file dates. Note, you must not rename, or otherwise alter the date stamps on these files.
  2. use find with -newer <lowbarfile> ! -newer <highbarfile> to select files for transfer, piping to rsync like your reference question.
  3. every week (or every night), re-rsync the whole directory to make sure nothing was missed. Get an email log of files transferred this way so you can see if problems are occurring with the earlier steps.

This isn't as awesome a solution as inotifywatch, but it also doesn't break after 8000 directories and your hierarchy appears to use up to 256+65536 dirs.

Andrew Domaszek
  • 5,163
  • 1
  • 15
  • 27
  • This would only work for *pushing* files with rsync, right - I couldn't use this to pull recent files? I know the question says I'm using "push"... – Barry Pitman Nov 27 '14 at 11:01
  • Right. There's nothing to say you couldn't run the push script remotely using `ssh `, but you need to be on the origin side to generate the updated file list with find to pass to rsync. The list itself could be passed to a pulling rsync, but you have to be on the origin to generate it. – Andrew Domaszek Nov 27 '14 at 14:46
1

For each run rsync needs to establish a complete listing of both the local and remote directory structure and compute the differences, before it has determined which files are newly created and sending those new files over. That is what is "expensive".

You haven't mentioned what the OS of the file-server is, but on Linux you can use something like inotofywatch to generate an alert on each filesystem event that creates or modifies a file, and use that event as a input to copy the new files. Your tiered directory structure makes inotifywatch somewhat expensive though.

On Windows you have DFSR which does roughly the name, it also plugs in the file system layer and is even more intelligent in the regard only the modified part of a file is replicated, instead of the whole file.

HBruijn
  • 77,029
  • 24
  • 135
  • 201
1

You could run rsync with -e "ssh -C" thereby compressing the ssh tunnel instead of only the data as it does when running with -z. Or connecting thought a vpn which compresses the traffic (openvpn can do this).

Jorge Nerín
  • 1,138
  • 8
  • 8
  • I tried this and there was no improvement. The bandwidth is used up by the sending filenames, and since the filenames are sha256 hashes, I don't think they'll compress well at all. – Barry Pitman Nov 27 '14 at 10:54
  • You might be right, then I taking into account the restriction you have about files not being deleted & modified I would suggest anything using the inotify or fsnotify like https://code.google.com/p/lsyncd/, test it with your dataset and please report back the outcome I would be glad to know it. – Jorge Nerín Nov 27 '14 at 13:42
  • I'm surprised that ASCII text (which essentially uses only 5 bits) doesn't compress more. You might try the benchmarks again to verify your methods, compare "ssh -C" vs "-z" vs neither, and possibly capturing packets to verify. Also... how are you measuring the bandwidth used? – TomOnTime Nov 27 '14 at 15:09