How to update millions of records in MySql?

Question

I have two tables tableA and tableB. tableA has 2 Million records and tableB has over 10 millions records. tableA has more than thirty columns whereas tableB has only two column. I need to update a column in tableA from tableB by joining both tables.

UPDATE tableA a 
INNER JOIN tableB b  ON a.colA=b.colA
 SET a.colB= b.colB 

colA in both table has been indexed.

Now when I execute the query it takes hours. Honestly I never saw it completed and max i have waited is 5 hours. Is their any way to complete this query within 20-30 minutes. What approach should I take.

EXPLAIN on SQL Query

"id" "_type" "table" "type" "possible_" "key" "key_len"   "ref"   "rows" "Extra"
"1" "SIMPLE" "a"     "ALL"  "INDX_DESC" \N    \N          \N   "2392270"  "Using where"
"1" "SIMPLE" "b"     "ref"  "indx_desc" "indx_desc" "133" "cis.a.desc" "1"  "Using where"

Answer 1

Your UPDATE operation is performing a single transaction on ten million rows of a large table. (The DBMS holds enough data to roll back the entire UPDATE query if it does not complete for any reason.) A transaction of that size is slow for your server to handle.

When you process entire tables, the operation can't use indexes as well as it can when it has highly selective WHERE clauses.

A few things to try:

1) Don't update rows unless they need it. Skip the rows that already have the correct value. If most rows already have the correct value this will make your update much faster.

    UPDATE tableA a 
INNER JOIN tableB b  ON a.colA=b.colA
       SET a.colB = b.colB
     WHERE a.colB <> b.colB 

2) Do the update in chunks of a few thousand rows, and repeat the update operation until the whole table is updated. I guess tableA contains an id column. You can use it to organize the chunks of rows to update.

    UPDATE tableA a 
INNER JOIN tableB b  ON a.colA=b.colA
       SET a.colB = b.colB
     WHERE a.id IN  (
             SELECT a.id
               FROM tableA
              INNER JOIN tableB ON a.colA = b.colA
              WHERE a.colB <> b.colB
              LIMIT 5000
      ) 

The subquery finds the id values of 5000 rows that haven't yet been updated, and the UPDATE query updates them. Repeat this query until it changes no rows, and you're done. This makes things faster because the server must only handle smaller transactions.

3) Don't do the update at all. Instead, whenever you need to retrieve your colB value, simply join to tableB in your select query.

Answer 2

Chunking is the right way to go. However, chunk on the PRIMARY KEY of tableA.

I suggest only 1000 rows at a time.

Follow the tips given here

Did you say that the PK of tableA is a varchar? No problem. See the second flavor of code in that link; it uses ORDER BY id LIMIT 1000,1 to find the end of the next chunk, regardless of the datatype of id (the PK).

Answer 3

Hi i am not sure but you can do by cron job. process: in table tableA you need to add one more field (for example) is_update set its default value is 0, set the cron job every min. when cron is working: for example it pick first time 10000 record having is_update field 0 value and update records and set is_update is1, in 2nd time its pick next 10000 have is_update 0 and so on... Hope this will help to you.

Answer 4

For updating around 70 million records of a single MySQL table, I wrote a stored procedure to update the table in chunks of 5000. Took approximately 3 hours to complete.

DELIMITER $$
DROP PROCEDURE IF EXISTS update_multiple_example_proc$$
CREATE PROCEDURE update_multiple_example_proc()
BEGIN
DECLARE x  bigint;

SET x = 1;

WHILE x  <= <MAX_PRIMARY_KEY_TO_REACH> DO
UPDATE tableA A
   JOIN tableB B
   ON A.col1 = B.col1
SET A.col2_to_be_updated = B.col2_to_be_updated where A.id between x and x+5000 ;
SET  x = x + 5000;
END WHILE;

END$$
DELIMITER ;

Answer 5

Look at oak-chunk-update tool. It is one of the best tool if you want to update billion of rows too ;)