1

In my app in heroku I have a task that needs more than 30 seconds to execute and I can't increase the execution time because heroku won't allow that. I take data from an API in the front-end and then send it using AJAX to the server to insert this data to the database. This process in the server takes more than 30 seconds to complete and it gives me a timeout error.

I created the Procfile as below : worker: cd ~/www/ && ~/php/bin/php worker.php

The file worker.php takes the data sent using AJAX and then inserts it into the database.

I enabled the worker in the heroku dashboard

What I want to achieve is start this worker only when I click a button that start the ajax call. I don't want to use heroku scheduler because this script will only be run once a month or whenever the client wants to add new products. Also if I use the scheduler I don't know how to send the data from the API to the server.

If this isn't possible is there a way to force heroku to increase the execution time?

Thank You

Alfred
  • 59
  • 11
  • ... why do you need it as background worker while indeed it is only being executed when an Ajax POST comes in... like how normally webhosting is? – Nick Li May 01 '17 at 14:10
  • @Nick Li because the whole app freezes until the insertion to the database is finished. I think as a background worker this would not happen – Alfred May 01 '17 at 14:13
  • I doubt that is the problem, even if you insert a 100k row it should not take 30 seconds. And it is more than 30 seconds to execute from a single Ajax, the scale of your script probably justify to have a private server instead. What are you doing with your script apart from inserting data? – Nick Li May 01 '17 at 14:18
  • Alright, I exaggerated a bit ... but you probably need to optimise how you insert into MySQL to reduce execution time, check this other post out http://stackoverflow.com/questions/19682414/how-can-mysql-insert-millions-records-faster – Nick Li May 01 '17 at 14:25
  • @NickLi This is the exact activity of the script. I send an array with around 7000 objects inside to the server. Then I loop inside this array and for each object id I check if that value already exist in the database, if it does I update I update the whole row with the new values, if it doesn't I insert it as a new row. So it's not exactly just inserting – Alfred May 01 '17 at 14:28

1 Answers1

0

What you need is to optimise your sql queries to reduce the execution time instead.

According the description of your comment, the action you trying to do is "UPSERT" - update the existing row or insert if not exist.

Try google sql upsert and you will find more information. NOTE: upsert can means several different situation and each need to tackle differently. read this article which helped me much on sql upsert

Other consideration is drop unnecessary indexes for the table which makes insert even slower as it re-index for every insertion if not bundled. If you are still in development that applies to unnecessary foreign key as well.

Another side note, in another post it said normally sql can handle ~40k insert/minutes, so if you are inserting more than 15-20k, which is about 30s you probably should consider a private server or maybe upgraded heroku allow more resource or execution time that may suit your need.

Nick Li
  • 312
  • 2
  • 12