10

We have a simple web application running on a virtual machine that saves its data in a MySQL 5.5 database with the InnoDB engine. Everything worked fine for around three years, but suddenly it became extremely slow.

For example, I have a very simple table holding addresses:

CREATE TABLE `addresses` (
  `address_id` int(11) NOT NULL AUTO_INCREMENT,
  `name` varchar(64) CHARACTER SET latin1 NOT NULL,
  `firstname` varchar(64) CHARACTER SET latin1 NOT NULL,
  `street` varchar(64) CHARACTER SET latin1 NOT NULL,
  `housenumber` varchar(16) CHARACTER SET latin1 NOT NULL,
  `zip` varchar(5) CHARACTER SET latin1 NOT NULL,
  `city` varchar(64) CHARACTER SET latin1 NOT NULL,
  `email` varchar(64) CHARACTER SET latin1 NOT NULL,
  `phone` varchar(16) CHARACTER SET latin1 NOT NULL,
  `birthdate` date NOT NULL,
  PRIMARY KEY (`address_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin

This table hold around 800 entries which is really not much. But running the query

SELECT * FROM addresses

for test purposes, it seems to never finish. I checked this with the mysql CLI on the server itself: It outputs some rows of the table and then waits very long until it outputs the next rows.

So maybe it is a problem in the data sending phase, but I am not sure.

The VM has 2GB of RAM and only 320MB are used. The CPU also runs at very low 1 to 2%. mytop does not show any other queries that are blocking the server. The IT admin said that they didn't change anything at the hardware side.

I already tried some thing like restarting the database server, restarting the virtual machine. Nothing helped.

edit:

EXPLAIN SELECT * FROM addresses

gives me this result:

+----+-------------+-----------+------+---------------+------+---------+------+------+-------+
| id | select_type | table     | type | possible_keys | key  | key_len | ref  | rows | Extra |
+----+-------------+-----------+------+---------------+------+---------+------+------+-------+
|  1 | SIMPLE      | addresses | ALL  | NULL          | NULL | NULL    | NULL |  793 |       |
+----+-------------+-----------+------+---------------+------+---------+------+------+-------+
1 row in set (0.00 sec)
tabb
  • 131
  • 2
  • 6
  • Virtual Box, Xen or something else? How is the host disk set up? are there other VM's on the same box running slow also? Answering those questions might reveal the answer – hookenz Nov 10 '16 at 20:00
  • I am not the owner of the hypervisor, so I cannot really answer this. I already asked the admin of the hypervisor if they know of any recent changes, but he said no. – tabb Nov 10 '16 at 20:13
  • if you can, shut down mysql and run some disk and memory benchmark. Doesn't have to be complicated. The idea is to just isolate if it's an issue with mysql, perhaps index problem as indicated in the answer below, or more widespread. – hookenz Nov 10 '16 at 20:18
  • Good point, I experienced it to probably not being an mysql problem. Running `mysql -u username -ppassword mydb -e 'SELECT * FROM addresses` is outputting slow, but appending ` > test.txt`, it runs very quick. Now this would probably be a different question!? How could I investigate on this? – tabb Nov 10 '16 at 20:31
  • Contact the owner of the hypervisor and ask them to check the logs for any errors. Specifically disk errors. Tell him your symptoms. Backup now. – hookenz Nov 10 '16 at 21:35

2 Answers2

13

If the cpu load is low then this indicates that there are no problems with missing indexes, if that was the case the query would just need take more cpu and disk access. Also you said it worked fine for 3 years.

Did you check the general disk access speed (specifically on the partition where the database is located)? E.g. using dd like here. What you're describing sounds like a dead disk or half-dead raid. Got backups i hope?

user3648611
  • 141
  • 3
9

You can trying a couple things,

  1. Do you have indexes setup?

Indexing makes it possible to quickly find records without doing a full table scan first, cuts execution times dramatically.

CREATE INDEX idx_name ON addresses(name);
  1. Before running the query use the EXPLAIN keyword first,

When used in front of a SELECT query, it will describe how MySQL intends to execute the query and the number of rows it will need to process before it finishes.

  1. Make some changes to your mysql.ini, if its a VM increase the RAM and configure your mysql.ini to see if the performance increases.

There are many MySQL optimizers out there that can guide you.

Help this helps

Anthony Fornito
  • 9,546
  • 1
  • 34
  • 124
  • Okay, but creating an index on a quite small table (<800 rows) seems to not be as helpful as I would need. It takes over a minute to finish the query quoted above. A full table scan of such a small table shouldn't take so long. – tabb Nov 10 '16 at 19:48
  • So... did you add the indexes? If you are not sure what the problem is I would start basic and work my way down. Adding indexes will only increase performance if done correctly. – Anthony Fornito Nov 10 '16 at 19:57
  • Yes, I added an index on the name column. But my above query doesn't even have any restricting WHERE clause, so it will have to read the whole table and print it out. Added the index didn't help. – tabb Nov 10 '16 at 20:08
  • I added the result of the EXPLAIN query above. This looks fine to me, but the performance is still very low. I cannot increase the RAM as I am not the admin of the virtualization manager. But `htop` shows that only 307MB out of 2050MB RAM are used. – tabb Nov 10 '16 at 20:10
  • About indexes you really need to think about what columns to index, you don't want to just index everything, index the ones that have the largest amount, i am making some wild assumptions on this but i would start with `name` and 'firstname'. Second are you sure you did the indexing correctly? possible_keys: NULL if the column is NULL, it indicates no relevant indexes could be found. – Anthony Fornito Nov 10 '16 at 20:15
  • If you have that much RAM available edit you my.ini restart the service see if makes a difference. – Anthony Fornito Nov 10 '16 at 20:16
  • please see my comment above, it seems to not only be a MySQL-related problem ... – tabb Nov 10 '16 at 20:42
  • Ok i read the comment " But htop shows that only 307MB out of 2050MB RAM are used" Is that what mysql us configured to use? Or is that what the system has available to it? – Anthony Fornito Nov 10 '16 at 20:44
  • @AnthonyFornito , the test query was `SELECT *`, so no indices help there, so the problem is elsewhere. – Tero Kilkanen Nov 10 '16 at 23:07