1

I have to copy (periodically) a MSSQL backup file large 500 GB to another pc (windows or linux) within lan.

The normal copy (or xcopy, robocoy, teracopy) command to a network share (or external usb disk) takes a lot of time (about 12 hours) and the server with the backup file become very slow (because there is only one RAID SAS controller).

An "idea" maybe to split the backup file into smaller parts (like torrent) and copy these parts concurrently, but I don't know how to do it.

My primary target is to copy without impact the source server.

Any idea or tool suggested will be appreciated.

GregD
  • 8,713
  • 1
  • 24
  • 36
lg.
  • 4,649
  • 3
  • 21
  • 20
  • There's a good chance that copying smaller parts at the same time will reduce performance as the machine has to start moving the heads more often to chase down multiple sections at the same time (not to mention the CPU and network overhead from multitasking). – Brian Knoblauch Sep 03 '10 at 15:05

6 Answers6

3

With windows, unfortunately, your options are a bit more limited. You could simply try setting up a FTP server (don't use the IIS one) and set a max bandwidth. This should effectively lower the impact on the server.

In all honesty however, If you're running into a noticeable slow-down when doing a single-threaded copy of a single file... I suspect that there are other issues at play. If you're talking about copying from SAS -> USB (or vise-versa) and the lag is happening... There is probably some issue with the USB stack. The USB stack shouldn't affect system performance that drastically. If you are running into the lag when doing ANYTHING from the SAS volume... perhaps the firmware of your raid controller is b0rked... or you're using the wrong drivers... (or in some cases... a REALLY cheap SAS controller). It can also be the result of some anti-virus program (or multiple programs overlapping) causing the machine to work 4x as hard as necessary. (as a rule... only use 1 anti-virus program. More will just make your machine work harder for no reason.) Even having your volume fragmented all over the place can cause the symptoms you're talking about.

I would first try to fix the problem rather than simply work around it.

TheCompWiz
  • 7,409
  • 17
  • 23
  • The SAS controller works well, the problem is that the server has 6 disks SAS (RAID1 and RAID0) managed by only one controller; and the RAID1 volumes are used as data partition for a database server with high traffic; and the large file to copy is on RAID0 partition. – lg. Sep 03 '10 at 14:16
  • I will test the ftp solution. – lg. Sep 03 '10 at 14:16
  • Seconded. Also, if the problems are related to IO on your disk/controller, splitting the file on that volume will cause the same performance problems, and then copying multiple smaller files from that volume will probably cause even worse performance problems. – mfinni Sep 03 '10 at 14:17
  • Crap. Seriously. I have ONE raid controller running 13 discs in a computer and I can totally copy nearly a gigabyte of data per second without the LAN getting slow (on the discs). 1gb network is only 100mb/s and should not overload a high end RAID 1 or RAID 0. – TomTom Sep 03 '10 at 15:02
  • SOMETHING is wrong here. – TomTom Sep 03 '10 at 15:02
  • It is not slow the network, but become slow the server form where I copy the file. – lg. Sep 03 '10 at 15:09
  • I have several setups where I'm using 16 disk on 1 controller with NONE of the symptoms you're describing. If it's a cheap software-raid controller... I can understand to a degree what you're saying. If it's doing all the raid0/1 in hardware, that should not be the source of the lag. Consequently, what is the point of having RAID0... if you don't get the performance benefits due to a card that can't handle it? – TheCompWiz Sep 03 '10 at 15:57
  • I accepted you answer because I tried to transfer the file by FTP, and it seems the best solution for me (lower source server impact). – lg. Sep 13 '10 at 13:13
2

By your description, the server cannot cope with the I/O load. I would be inclined to think that's due to some other fault as TheCompWiz said. Until/unless you fix/alleviate the cause of the problem (you think it's insufficient spindles or too slow a RAID controller) you will not get it copied across any quicker.

As your foremost intent is to copy it without impacting the server, look at throttling bandwidth/copy speed, i.e. deliberately making it run slower. robocopy has an "inter-packet gap" option robocopy /ipg:N (where N is some number) which can be used to limit speed. http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx. I understand teracopy does too.

To clarify do you need to copy this file only once, or are you needing to copy a backup file periodically on a schedule? If it's periodically, for backup purposes, then as Tom said, copying only what has changed from one 500GB file to the next 'version' will save a great deal of time and put less load on the server. Perhaps if you explained your overall intent rather than the bit you're currently stuck you might get some useful ideas.

Norky
  • 849
  • 4
  • 14
0

the only way to really make this go alot faster is to go to some kind of block level solution that would only ever copy the deltas.

SpacemanSpiff
  • 8,753
  • 1
  • 24
  • 35
0

Why put lots of individual files into one big one then try to split it up again?

Backup the individual files.

symcbean
  • 21,009
  • 1
  • 31
  • 52
0

Have you monitored memory usage on the windows server while doing the copy? I wonder if you're running into the "large files" issue.

Have a look at Windows Server 2008 x64, Large File Transfers, and Memory Usage (and some of the other pages linked from there)

Community
  • 1
Chris_K
  • 3,444
  • 6
  • 43
  • 45
0

Other solutions: Use differential or transactional backup instead of full backup. - This will cut down on the amount of data you will need to transfer.

CDP: I use R1soft http://www.r1soft.com/ This software has saved my ass so many times.

Kyle Buser
  • 246
  • 1
  • 9