0

We have a business, where we send API request to generate digital products. We have 2 jobs, one order based and one cron-job to process open orders.

I don't know where i should search the Problem.

Problem is; the Cronjob double processed now 3 times (on 3 different days) products. Example: Our Shop asks with a simple JSON POST the other system to get the Products by API. This is initiated by the Cronjob. The other System always makes, what the request incoming want.

If the Shop asks for 1 product, the system ordered two. Two similar requests are made to the API.

I don't know if this Problem is coming from;

  • the store (where the Cron Job Script is started)
  • the System
  • somewhere else

It is suspicious, that not the whole Job is done twice, it was only for some orders and some products. And not always. Does any one had a some Problem or a probably solution for that?

We already have some Security checks to prevent that, but it looks like it doesn't help. (The double processing is very bad, the products are like money (giftcards), if a customer get's to much of the digital cards and he redeem it immediatly, the money is lost).

domof
  • 1
  • Sounds like a race condition... how frequent is the job run? how long does it need for processing? maybe they're cascading..? – Honk der Hase Nov 30 '20 at 22:15
  • Thanks for the Feedback. The Cron runs every Minute (* * * * *), daily, it never stops. It depends on, how much orders are processed. Mostly, an order is processed within 1-4 seconds and the complete job under one Minute (also on my Problematic days, it didn't took longer). But also for this, we have a security, if the job starts and he picks up an order, it writes the order id in a table to lock the order. if it takes more than 1 period (1minute) and next job would check that the order is locked and it wouldn't pick it up. So that isn't the Problem :/ – domof Dec 01 '20 at 13:48
  • Still enough potential for a race condition, the database can be the bottleneck... if queries are queued due to other long-running queries, the update of a record could be delayed... and if this gap is larger than your cron interval, the info "record in progress" could still be in the queue, while the new job starts. the new job then adds its own queries to the queue, meanwhile the previous job could finish.. but since the next job is already queued, it will still think the record it processes hasn't been processed yet. – Honk der Hase Dec 01 '20 at 16:54
  • I'd suggest you'd add some logging (file-based!) and log date+time and the record (PK) on 4 events: cron starts, query start, query finish, cron finish. If I'm right, you should see the same PK started twice (cron) before any end (cron). Most of the time, it should read "cron start, query start, query end, cron end" with the same PK, but if it appears in other orders, it would be a strong indication for a race condition. – Honk der Hase Dec 01 '20 at 17:11
  • We have Logs, we see all in there. The start, the end of the Process. We also see that, there is a line twice for this specific request. So the Script/server did it twice and not only once as it should like; request 1 code, if this is fired twice we get 2 instead of 1 and that's the Problem. Wer checked the cron itself a lot, we tried to run the Job like 3 times (pressing the start button 2 times and hit the URL from the browser) and it worked good in this way (it didn't processed it more times, the Lock worked) – domof Dec 02 '20 at 08:56

0 Answers0