2

I have the schedule job of backing large files in Windows Server and it takes almost 2 hours to finish.

For now, I only use copy command in the .bat and copy the large files to another folder.

I'm wondering whether there is more efficient way than the copy command to backup the large files(approximately 90GB). For example, write an c++ program to copy the files rather than the copy command.

Many thanks.

macemers
  • 131
  • 5
  • 2
    Do all these files change every day? Have you considered an incremental backup using something like rsync? – Thilo Jul 20 '12 at 03:47
  • What types of files are these? Are these large binary files like pictures and videos? Or are these large text files like csv files, spreadsheets, etc? Is each file you are backing up approx. 90GB or is the entire backup approx. 90GB? –  Jul 20 '12 at 03:52

4 Answers4

2

The copy command you're using is likely already written in C++ or C, or even with lower-level routines, so you won't really see any speed gains by simply using existing copy functions or methods in those languages.

Alex Reynolds
  • 453
  • 2
  • 9
  • 22
1

You will be disk bound. Most likely the language/mechanism will be the same in the end (system calls). The fastest way is to upgrade disks to higher RPM and/or use a striped RAID configuration.

But make sure you are copying between two seperate physical drives. (i.e. reading from one and writing to the other).

If possible, just save the changes (deltas), not the whole file, perhaps every week doing a full backup. Also if the files are well suited to compression (e.g plain text), then compressing prior to writing to disk will be much faster.

0

If you need to preserve file ACLs, consider xcopy or robocopy. I used it to mirror almost 600 GB of data from 2 different disk (400GB + 200GB) to a backup disk. Daily changes are almost 2GB (mostly small files) and finished copying in 45 minutes. The initial mirroring will take some time proportional to your data.

If need not preserve the ACLs, Terracopy will be faster

Hendry Leo
  • 24
  • 3
0

If the problem is the two hour downtime window while you back up the files, you might:

1. Move the files elsewhere on the same disk/array.

Provided you dont cross onto another disk, the move should be almost immediate. Requires diskspace. After the move, you should be able to restart the process that writes to these files, with just a few minutes of lag. You back up the moved files.

2. Rename the files

The files remain in the same spot, but have a different name.

You can rename the files. Requires diskspace. After the move, you should be able to restart the process that writes to these files, with just a few minutes of lag. You back up the renamed files.

3. Archive the files before backing them up

If these large files can be compressed, you may gain both time in backing them up, and you'll use less tape, for a double savings. Be sure to choose a good archiving program.

RobW
  • 2,806
  • 1
  • 19
  • 22