0

Whenever I run du -sh or rsync I get my disk utilisation at 100%. This causes mysql slow queries to appear, which usually causes my site going down.

Any tips or tricks running these to commands somehow, so the priority for disk access stays with Mysql and other core services ?

Coops
  • 6,055
  • 1
  • 34
  • 54
Katafalkas
  • 523
  • 2
  • 8
  • 20

4 Answers4

6

Yes, with ionice:

ionice -c 3 <pid>

where <pid> is the process ID of the process you want to slow down. This will set the I/O class of this program to idle, which means it will get IO time when nothing else is asking for it.

Sven
  • 98,649
  • 14
  • 180
  • 226
  • 1
    i think you can also do `command & ionice -c 3 $!` to do it in one go. need to check that – Sirex Mar 09 '12 at 11:56
3

You can use ionice -c3 as a prefix for your commands, that way those commands will run within idle I/O class, meaning they will only get I/O time when no other process needs it. This requires your system to use CFQ I/O scheduler, though. It's the default I/O scheduler in most Linux distributions, nowadays.

However, that kind of behaviour is usually symptom about something else. Is your disk nearly full? That causes slowdowns for systems having lots of writes, since everything tends to get fragmented. Or do you have tens of millions of small files around?

Janne Pikkarainen
  • 31,852
  • 4
  • 58
  • 81
  • Was going to say just this. It will probably be worth your while to have a look at your storage systems. If you can't manage the storage as it's being used it is probably indicative of it being maxed out, and so highly likely to be a bottleneck in your platform. – Coops Mar 08 '12 at 13:42
  • I will try doing so. I think I ll have to change my rsync command a little. Originally I was running rsync from backup machine, and it used to lock my production machine. I assume it is possible to rewrite rsync so i run it from production machine with "ionice -c3" to the backup machine. – Katafalkas Mar 09 '12 at 07:45
1

Do you have a kernel that has PREEMPT turned on? There are various kernel PREEMPT settings that can be turned on that will avoid lockups, such as during heavy disk load (perhaps at some cost of performance of server apps). It depends on your distribution how much is turned on by default.

In Debian backports there now is for example linux-image-rt-amd64 which includes the PREEMPT_RT realtime patch set. This patch set may help even more, however it may have its own problems...

Run something like this to see what is set:

# grep PREEMPT /boot/config-2.6.32-5-amd64    <-- in case of a Debian stable amd64 kernel

Result:

# CONFIG_TREE_PREEMPT_RCU is not set
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set

You may want to have it set to something more aggressive. Look around in the kernel documentation that comes with the source of your kernel to see what would work for you. And check what kernels your distribution provides that may have more aggressive PREEMPT settings. Those could be meant for desktop usage and/or real time usage. But that doesn't mean it won't work on a server. Provided they're part of the standard distro those should get the usual security fixes.

aseq
  • 4,610
  • 1
  • 24
  • 48
  • Again, PREEMPT has to do with the CPU, not IO, so won't help here. These settings are to prevent the kernel from using too much cpu on behalf of a process all at once, thus preventing other processes from getting any for some period of time ( a few milliseconds ). They are only needed if you have a program that is interacting with hardware that has specialized low latency requirements where a few millisecond delay would cause problems. – psusi Mar 09 '12 at 15:04
0

In addition to ionice(1) rsync might be pushing blocks needed by mysql out from disk cache. You can patch rsync with fadvise support to avoid putting the stuff read by rsync into cache.

ptman
  • 28,394
  • 2
  • 30
  • 45