8

I need to move a large file (corrupt MySQL table ~40GB) onto a seperate server in order to repair it. (When trying to repair on my production server, it quickly killed the server).

In order to do this, I want to rsync the .frm, .MYI and .MYD files from my production server to a cloud server.

I am copying the files from /var/lib/mysql/{database}/ to /home/{myuser} so that I don't need to enable root access for the rsync command and be 100% sure the database file isn't in use (it shouldn't be written to or read from, but obviously I don't want to shut down my production database to make sure).

The first file I tried to copy was around 10GB. I am transfering from one part of my production server to the other, i.e. to the same array of disks.

Unfortunately the copy command "cp filename newfilename" took so much resources it brought the server to a standstill.

How can I use less resources when copying the file to a different directory? (It doesn't really matter how long it takes).

Assuming I manage to do this, what resource usage can I then expect when rsyncing the file to the cloud?

Can anyone suggest a better way to do this? I am quickly running out of disk space so need to get this table repaired and archived ASAP.

rzr
  • 259
  • 2
  • 6
Jon M
  • 457
  • 1
  • 6
  • 11

4 Answers4

10

Two choices besides the rsync bandwidth limitation:

  • ionice -c 3 cp foo bar
  • buffer -u 150 -m 16m -s 100m -p 75 -i foo -o bar

ionice will interface with the I/O scheduler. buffer is a circular buffer meant to aid character devices in being more efficient, but the -u 150 will pause 150 microseconds between writes which, according to the manual, may be enough to give a disk room to breathe.

Both ionice and buffer are available in a stock Ubuntu build. iotop is handy if you happen to have CONFIG_TASK_DELAY_ACCT configured in your kernel, but my Ubuntu box did not which severely limits the usability of the command. I already know which command is drowning my hard drive, I just want to give it some breathing room.

Additionally, while the copy is in progress, look at the output of iostat -x 1 (usually in the sysstat package) and see that the %busy field for your device is 90% or less during the copy. If it is at 99-100%, then you are starving other processes for I/O.

zerolagtime
  • 1,428
  • 9
  • 10
6

use rsync with the --bwlimit=KBPS switch (limit I/O bandwidth; KBytes per second). play around with a smaller file and try to find optimal mix between transfer speed and system usage. Monitor in second shell with "vmstat 1"

shakalandy
  • 778
  • 4
  • 10
  • I will certainly try this to limit the resources of the RSYNC command, but at the moment I am just trying to copy a file to another location on my server via the "cp" command. – Jon M Apr 11 '11 at 17:09
  • you could also use rsync to copy the file local and limit the bandwith with the --bwlimit switch. – shakalandy Apr 11 '11 at 21:17
  • `rsync` will copy files locally. Think `rsync --bwlimit=28000 foo.frm /bar/foo.frm` and then watch it with `iostat`. I'm not sure why `vmstat` would tell you if you were saturating a drive. Look for %busy in `iostat`. – zerolagtime Apr 12 '11 at 03:43
  • well, vmstat shows your IO on block devices. In addition you can see anything else that is important to machine usage (cpu, user processes, cpu wait, swap,...) – shakalandy Apr 12 '11 at 07:01
5

Have you tried nice -n10 prefixed to the command ?

10 is default value. The range goes from -20 (highest priority) to 19 (lowest).

Jonathan Ross
  • 2,183
  • 11
  • 14
  • @Jonathan My understanding is that this will limit the CPU useage only, not Disk I/O AND RAM: http://www.linux.com/archive/feed/58638 – Jon M Apr 11 '11 at 17:08
  • Yes but it's a start, in terms of reducing one of the bottlenecks - it might be enough to stop the machine hanging. – Jonathan Ross Apr 11 '11 at 17:20
  • Thanks @Jonathan Ross, according to this: http://linux.die.net/man/1/ionice "Programs inherit the CPU nice setting for io priorities.", so it might just work, but I am researching a combination of "nice" and "ionice" – Jon M Apr 11 '11 at 17:25
  • Good thinking. I was always start with `nice` out of habit mostly. `iotop` is also your friend. – Jonathan Ross Apr 11 '11 at 17:34
  • 3
    The final command I ran was `nice -n 20 ionice -c 3 cp /var/lib/mysql/{database}/{table}.MYI /home/{USER}/` This did increase load on my server slowing it down a little, but the website was available for the duration of the transfer – Jon M Apr 12 '11 at 09:05
  • Glad it worked out for you. I'll be sure to remember `ionice` too ! – Jonathan Ross Apr 12 '11 at 09:11
0

One alternative is :

scp -l ${KBPS} ${src} ${dest}

But I dont think it will work if ${src} is a growing file ... can you suggest a way to copy and wait source to be closed ...

rzr
  • 259
  • 2
  • 6