0

We're using MySQL and have an ~1.1 million record InnoDB table where the entire table has to be replaced with data from a CSV twice per day.

What I've been doing up until now is truncating the table, then using mysqlimport to repopulate the table from the fresh CSV twice a day. This results in around 160 seconds of downtime, which I was initially told was fine, but that is no longer the case.

The only way I can think of to do this is to have a staging table and a production table. Live queries will go to production, and when I need to repopulate the db, I'll truncate staging, then mysqlimport the CSV into it, then after that is done I will swap the names so that staging will have the old database and production will contain the freshly imported CSV. I guess I would need an intermediate name, like production becomes production2, the staging becomes production, and then production2 becomes staging.

Does anyone have an alternate solution? The table needs to be imported as quickly as possible which is why I'm using mysqlimport, and the downtime must be extremely minimal.

tiredofcoding
  • 25
  • 5

2 Answers2

3

I think you already have the optimal solution. Just make sure you do an atomic rename, i.e. "rename table production to production2, staging to production". This way it will be transparent to the application/users. One thing to watch out for is if there are foreign keys that reference this table, because the keys will still reference the renamed old table. But I guess since you are able to truncate it before without any issue, you don't have any in place.

EDIT: I'm quite new to commenting on StackOverflow so I don't know if this is the best place to do so, but I just want to comment on the answer posted by eggyal - TRUNCATE causes an implicit commit so the proposed solution won't work as described (i.e. the changes will be seen to the user/application even before the new data is loaded).

Johann Tagle
  • 106
  • 6
  • Comment under eggyal's message, rather than diluting your answer with a comment from another answer! HTH – Rippo May 02 '12 at 08:29
  • Thanks Rippo, but I could not find any link to "comment" link on eggyal's message. Just found out now it's because I don't have the minimum 50 reputation yet to be able to comment on other posts. – Johann Tagle May 02 '12 at 09:14
  • Aha I see, here I will +1 you. Didn't realise comments needed 50 rep. – Rippo May 02 '12 at 09:16
  • I didn't realise `TRUNCATE` caused implict commit! One learns something new every day. Still, one can use `DELETE` instead... I will update my answer. – eggyal May 02 '12 at 09:50
0

Do the DELETE and LOAD DATA (that's all mysqlimport does anyway) operations in a transaction: MySQL will atomically do your staging process for you, without the effort.

START TRANSACTION;
DELETE FROM foo;
LOAD DATA INFILE '/path/to/bar' INTO TABLE foo;
COMMIT;

N.B.: As pointed out by @JohannTagle, TRUNCATE causes an implicit commit, so one needs to use DELETE instead.

eggyal
  • 122,705
  • 18
  • 212
  • 237