2

I have a dispatchable job to clear and rebuild the cache for a particular model.

The cache rebuilding takes about 10 seconds.

The job can - at times - get fired multiple times sequentially and I often end up with multiple queued jobs all doing the same cache clearing and rebuilding for the same model.

I'm looking for a way of deleting any of the same job (cache rebuilding) for that particular model just before dispatching the current job.

Ersoy
  • 8,816
  • 6
  • 34
  • 48
MonOve
  • 1,091
  • 13
  • 34

3 Answers3

1

There is no easy or %100 working way to remove duplicated job. Laravel's Redis queue driver uses sorted sets and lists for delayed/non-delayed jobs. This answer may provide some details about it. When you push the job into the queue then you can't remove it before it is processed. (you can try, but it is hard, the link has the answers).

What you may do is creating a control mechanism to prevent triggering cache invalidating job. Let's say before you push your job, you set a unique identifier. You may use Redis's SET command with EX and NX option.

  • Set key to hold the string value. If key already holds a value, it is overwritten, regardless of its type.
  • NX -- Only set the key if it does not already exist.
  • EX seconds -- Set the specified expire time, in seconds.

Before you push your job you execute the first command something like this;

127.0.0.1:6379> set mymodel:id:1 some-random-string-test EX 15 NX
OK
127.0.0.1:6379> ttl mymodel:id:1
(integer) 10
127.0.0.1:6379> get mymodel:id:1
"some-random-string-test"
127.0.0.1:6379> set mymodel:id:1 some-random-string-test-another EX 15 NX
(nil)
127.0.0.1:6379>

What you are doing is, you get the id of your model and create a key. You set your key with expire and set if not exists option. If the response is OK then you are setting this key for the first time in the given time interval(15 seconds in my example). If you get nil response it means that, the job for the given id is still locked. So you won't dispatch it. It will provide you a 15 seconds time window for each model that will prevent you to dispatch it.

--

If you can't(don't want) prevent the job to be triggered, then you can use the same command in a different way. Whenever you are about the dispatch the job you create a unique identifier and execute set mymodel:id:1 some-random-string-test EX 15 NX before dispatching it. But this time you will send identifier to the job's constructor too. At the beginning of your handle method, you will use redis's GET method to get the value of mymodel:id:1 and compare it with the identifier that you sent to the job's constructor. There will be only 1 matching identifier in 15 seconds, that means only one job will be "fully" processed. The other identifiers created for the same jobs will not be updating the the value of that redis key(NX option), therefore other job's identifier will not matched with the value of mymodel:id:1. That means they will be cancelled or not fully processed because they will not pass the condition.

Ersoy
  • 8,816
  • 6
  • 34
  • 48
0

Consider using setnx command on phpredis connection.
Note. This is different from the --tries option on php artisan queue:work command, and retries the failed job.

enter image description here

Majid Alaeinia
  • 962
  • 2
  • 11
  • 27
-1

Try using different queues for each of the jobs (f.e: queue per model) so you can check the queue size with Queue::size('queue_name') before dispatching the new job.

Professor
  • 858
  • 1
  • 9
  • 17
  • I would have to dynamically create queues, which lowers performance. Any other ideas? – MonOve Jul 27 '20 at 19:56
  • Another option that comes into my mind is to set some kind of semaphore (using the Cache facade) to avoid running multiple processes at the same time. So checking this Cache flag at the start of the job would work. – Professor Jul 27 '20 at 20:03