0

I have a schema which was originally set up with MyISAM. I had relationships between tables which were not defined as foreign keys, but I would link the table with joins, etc. There were obviously no rules for cascading deletes/updates, etc...

I re-created the same DB schema with InnoDB, but this time I put the foreign keys into the create statements and defined the on update/delete rules.

I'm loading about 200,000 records into the DB with PHP from a CSV file. It was originally taking a few minutes but since I swapped to InnoDB the PHP script times out after 10 minutes with only ~7% of the CSV file processed.

I'm tempted to switch back to MyISAM without the relationships, however a better solution (if possible) would be to use MyISAM while initially populating the database and then switch back to InnoDB afterwards and let MySQL fix up the relationships. I don't know if this is possible, but I assume that if it is, MySQL can fix up it's own internal data structures faster than PHP insertion one-by-one.

Another thing to consider is that approximately once a year, we will need to re-load the set of 200,000 records and identify which ones have changed. It probably wouldn't be possible to switch from one engine to the other whenever we needed to do this.

Any pointers?

Ozzah
  • 10,631
  • 16
  • 77
  • 116
  • Version of MySQL? Also, a direct answer: http://stackoverflow.com/questions/2167522/innodb-takes-over-an-hour-to-import-600mb-file-myisam-in-a-few-minutes/2167641#2167641 – Halil Özgür Aug 18 '11 at 07:13
  • Did you modify innodb variables to obtain better performance or is it all at default settings? – N.B. Aug 18 '11 at 07:59
  • Bear in mind that converting a MyISAM table into an InnoDB table is taking a lot of time as well, if many records are already in. – Pelle Aug 18 '11 at 08:05

2 Answers2

2

Difficult to say what you could be doing wrong without staring at your code, but a couple of things you could try:

  1. Run SET foreign_key_checks = 0; before you start loading and then SET foreign_key_checks = 1; after you're done loading to speed up the insert time.
  2. Ensure you're running your bulk inserts in a transaction so you aren't auto-committing each time.

See if those help.

Femi
  • 64,273
  • 8
  • 118
  • 148
2

Inserting data into the database can take up a long time if you're using one insert query per insert. It will speed up significantly when you use the format generated by mysqldump, like this:

INSERT INTO `table` (field1, field2, field3) 
VALUES
(value, value, value),
(value, value, value),
(value, value, value);

Also, if you'd like to insert it into myisam tables anyway, there's no problem either. Just make sure your foreign key integrity is right. After you've inserted all data, do this for all tables:

ALTER TABLE `table` ENGINE = innodb;
Friek
  • 1,533
  • 11
  • 13