1

We are hosting a set of websites on VM Cloud. These sites were previously on a couple of dedicated servers but to enhance performance, we transferred them onto a Cloud environment.

The Cloud has SSD storage but they are now saying that we have very high IOPS and are goign to degrade us if we do not do anything soon.

We have good PHP Websites but they are run without any Caching.

how do I start to debug this?

Sincerely,

Khuram

Edit: We have two servers where one holds the code while the other holds the databases. But both of them are showing very high (~95%) writes.

Khuram
  • 109
  • 3
  • 8
  • Please tell more about the OS you're running, what kind of software it is running (webserver), what kind of operations you're doing in PHP (file operations, MySQL?) and what kind of virtualization this is (with container based virtualization you can't do as much of debugging as you can with hardware virtualization). – gertvdijk Nov 21 '12 at 16:30
  • @gertvdijk The OS we are running is CentOS and the server is vcloud VM with 8GBRAM. The PHP websites are simple websites with regular reads and writes but here's the punch: IOPS reports tell that there is 95% writes and 5% reads and SSDs are hurting :-( – Khuram Nov 21 '12 at 16:37
  • Stop writing so much? – Michael Hampton Nov 21 '12 at 16:50
  • Yes, I am also stumped by this. We never had such stats in the older dedicated setup. – Khuram Nov 21 '12 at 16:55
  • 1
    95% writes and 5% reads is very unusual for a website and typically those stats are reversed. What kind of websites (e.g. blog, ecommerce, etc) are you running that writes that much? – daemonofchaos Nov 21 '12 at 17:08
  • They are affiliate websites. Tell me, if a MySQL query has joins then does each of those count as a separate IOPS? – Khuram Nov 21 '12 at 17:09
  • @Khuram A large amount of JOIN statements will cause more IOPS, but should not cause a 1:19 read/write ratio. Have you checked your query cache settings in MySQL? Use `mytop` or `mtop` to watch the transactions on the database. – tacotuesday Nov 21 '12 at 17:38
  • @nojak trying to install mytop but no package available in yum. – Khuram Nov 21 '12 at 18:09
  • @Khuram You'll need to add the EPEL repository. Instructions on how to do that can be found [HERE](http://www.cyberciti.biz/faq/rhel-fedora-centos-linux-enable-epel-repo/). – tacotuesday Nov 21 '12 at 18:18
  • @Khuram You can get `mtop` from the [RPMForge](http://wiki.centos.org/AdditionalResources/Repositories/RPMForge) repo. – tacotuesday Nov 21 '12 at 18:24

1 Answers1

2

It's a very long shot, this could be related to the atime filesystem attribute. If your application consists of a lot of files, there is a good chance that the writes to the disk are simply updates to the access time of the files on the filesystem.

Try to remount the document root partition with the noatime mount option and see how it goes. You could use the following command:

mount -o remount,noatime /dev/sda2

Where /dev/sda is the disk partition containing your document root.

melsayed
  • 1,122
  • 1
  • 6
  • 12
  • will this restart the whole system? I am not sure if I have the permission ( I am root but still its VM). We have a lot of production sites which I am uninclined to come in the way. – Khuram Nov 21 '12 at 17:08
  • No, this doesn't require a reboot. It can be done online with no downtime. – melsayed Nov 21 '12 at 17:35
  • And you have the permissions if you have root accces. – melsayed Nov 21 '12 at 17:36
  • I would also suggest to change `remount,noatime` line to `remount,noatime,discard`. But it's only for ext4 partitions. And make sure that kernel version >=2.6.33 –  Nov 21 '12 at 17:38
  • @melsayed its virtual setup so I do not see the partitions. mount: can't find /dev/sda2 in /etc/fstab or /etc/mtab – Khuram Nov 21 '12 at 17:44
  • Yes, sda2 is just an example. Run the `mount` command and provide it's output, please. – melsayed Nov 21 '12 at 17:47
  • @melsayed here's the mount info: http://pastebin.com/MnQT52jc – Khuram Nov 21 '12 at 18:02
  • Ok, try this: mount -o remount, noatime /dev/mapper/VolGroup-lv_root – melsayed Nov 21 '12 at 18:06
  • @melsayed mount: you must specify the filesystem type. Also its a shared cloud, so I am a little hesitant – Khuram Nov 21 '12 at 18:10
  • Try now, there was an additional space. mount -o remount,noatime /dev/mapper/VolGroup-lv_root – melsayed Nov 21 '12 at 18:12
  • @melsayed it ran without any print. What do I expect now? How do I check its working? – Khuram Nov 21 '12 at 18:21
  • You should check and see if the IOPS report is still high.. – melsayed Nov 21 '12 at 18:23
  • It does not seem to have helped as I am now monitoring with nmon, top and iotop. Same 0% reads and 95%+ writes. – Khuram Nov 22 '12 at 08:14