2

I have to copy some TB of data from one external drive to another (mostly ~400 MB files). The source is NTFS-formatted USB2 and the best read speed I can get from it is 7MB/s. The target is HFS+ USB3 and the write speed is considerably higher than that.

The NTFS filesystem isn't OK, and only NTFS-3g (slow, userspace) can read all the files in it, not even Windows can do it or I'd use that.

I'm using rsync -va /Volumes/source /Volumes/target to perform the copy.

Is there a way, in rsync or OS X, to cache the writes, so that the target doesn't need to be permanently writing at a slow speed for days? Something like filling 1GB or 2GB or even 4GB of RAM before writing? Thus the target drive would spend a few days doing

write - rest - rest - rest - rest - rest
write - rest - rest - rest - rest - rest
write - rest - rest - rest - rest - rest
...

instead of spending all those days continuously writing at a slow speed.

Another possibility, given that most files are large, would be to have rsync only write any given file after it's finished reading it, but I found no way to do that either.

Or is it better to just spend the whole time continuously writing at a slow speed?

entonio
  • 185
  • 1
  • 1
  • 8

2 Answers2

0

First, you're going to face an eternally long and difficult process reading from NTFS, without a good driver config.

Try http://www.macupdate.com/app/mac/26288/ntfs

That said, you're quite likely having issues with the original drive, if you're running that slowly.

There is no good way to change the cache process on official rsync

you might find that cp -rvn is a faster option in this situation.

frank ankersly
  • 199
  • 6
  • Thanks for the `cp` suggestion, it's often unjustly overlooked. I intend to reformat the source once I'm done with backing it up. I don't know if paragon's driver would be able to read it, I'd have run a `chkdsk` but I'm afraid it would mess it up even more. – entonio Nov 28 '15 at 23:24
0

I've found an answer of sorts: use a ramdisk as the temp dir for rsync, e.g.

rsync -va -T /Volumes/ramdisk /Volumes/source /Volumes/target

Make sure the temp dir actually exists, because otherwise rsync won't create it and will just read as into /dev/null, afaict (it may be that it will just instade use free RAM as a temp space, which would be great, but I doubt that, and I've too much RAM to wish to try that).

This will read each file complete to the ramdisk first, then write it to the target in one go. Since the files are large, the end result is more or less what I intended. It's not perfect because rsync stops reading the source while the target hasn't finished writing, but in this case that means the source won't be kept in uninterrupted activity for the whole transfer, which is fine by me. There isn't an overall slowdown because reading to the ramdisk is a bit faster than reading/writing into the target.

entonio
  • 185
  • 1
  • 1
  • 8
  • How to build a RamDisk? Answer: start here https://gist.github.com/nischu7/7194165 but note that the first few answers are wrong about the math. Sectors are not 2048 bytes but 512, so you need a larger RAM cache to get really big (4G) files into Cache. – Michael Tiemann Aug 05 '17 at 13:33