MySQL row lock and atomic updates

Question

I am building a "poor man's queuing system" using MySQL. It's a single table containing jobs that need to be executed (the table name is queue). I have several processes on multiple machines whose job it is to call the fetch_next2 sproc to get an item off of the queue.

The whole point of this procedure is to make sure that we never let 2 clients get the same job. I thought that by using the SELECT .. LIMIT 1 FOR UPDATE would allow me to lock a single row so that I could be sure it was only updated by 1 caller (updated such that it no longer fit the criteria of the SELECT being used to filter jobs that are "READY" to be processed).

Can anyone tell me what I'm doing wrong? I just had some instances where the same job was given to 2 different processes so I know it doesn't work properly. :)

CREATE DEFINER=`masteruser`@`%` PROCEDURE `fetch_next2`()
BEGIN
    SET @id = (SELECT q.Id FROM queue q WHERE q.State = 'READY' LIMIT 1 FOR UPDATE);

    UPDATE queue
    SET State = 'PROCESSING', Attempts = Attempts + 1
    WHERE Id = @id;

    SELECT Id, Payload
    FROM queue
    WHERE Id = @id;
END

Answer 1

Code for the answer:

CREATE DEFINER=`masteruser`@`%` PROCEDURE `fetch_next2`()
BEGIN
    SET @id := 0; 
    UPDATE queue SET State='PROCESSING', Id=(SELECT @id := Id) WHERE State='READY' LIMIT 1;

    #You can do an if @id!=0 here
    SELECT Id, Payload
    FROM queue
    WHERE Id = @id;
END

The problem with what you are doing is that there is no atomic grouping for the operations. You are using the SELECT ... FOR UPDATE syntax. The Docs say that it blocks "from reading the data in certain transaction isolation levels". But not all levels (I think). Between your first SELECT and UPDATE, another SELECT can occur from another thread. Are you using MyISAM or InnoDB? MyISAM might not support it.

The easiest way to make sure this works properly is to lock the table.

---

[Edit] The method I describe right here is more time consuming than using the Id=(SELECT @id := Id) method in the above code.

Another method would be to do the following:

1. Have a column that is normally set to 0.
2. Do an "UPDATE ... SET ColName=UNIQ_ID WHERE ColName=0 LIMIT 1. That will make sure only 1 process can update that row, and then get it via a SELECT afterwards. (UNIQ_ID is not a MySQL feature, just a variable)

If you need a unique ID, you can use a table with auto_increment just for that.

---

You can also kind of do this with transactions. If you start a transaction on a table, run UPDATE foobar SET LockVar=19 WHERE LockVar=0 LIMIT 1; from one thread, and do the exact same thing on another thread, the second thread will wait for the first thread to commit before it gets its row. That may end up being a complete table blocking operation though.