1

I am working on a data importing functionality for a website. I am using Laravel 5.1 on homestead, and have given the machine 4GB of RAM and two virtual cpus. For importing the data I used the Laravel Excel package, with its chuncking mechanism that basically breaks the data set in chunks and adds the processing to the queue. The workflow for this is as follows:

  • the user uploads the excel file;
  • a job is dispatched to handle the excel file;
  • this job then chunks the file and dispatches multiple other jobs to process the chunked data;
  • each of these chunks dispatches another job to do some background logic on the inserted data when they are done.

I have setup a queue named 'import' in supervisor for these processes. Along with this queue, I also have a default one (for sending mails and other, less intensive, stuff) and a low priority queue for dispatching jobs from other jobs. Here is my supervisor configuration:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=/usr/bin/php /home/vagrant/Code/iris/artisan queue:work --tries=1 --daemon --queue=default,import,background1
autostart=true
autorestart=true
user=vagrant
numprocs=8
redirect_stderr=true
stdout_logfile=/home/vagrant/Code/iris/storage/logs/worker.log

With smaller files this works well, but when I attempt to import a ~30k row document, the php processes spawned by supervisor run out of memory toward the end and in the laravel log I start seeing InvalidArgumentException: No handler registered for command [__PHP_Incomplete_Class] in Illuminate\Bus\Dispatcher (this happens especially when I run two imports in parallel or I attempt to download something through websockets) and I am really confused as to why this is happening. No one process exceeds the 512 MB limit as far as I can tell from using memory_get_usage(). Is this a garbage collector issue? Should I summon it manually?

And since I mentioned the websockets, I was also trying to create a separate queue (with higher priority) for handling web socket requests. I tried multiple supervisor configurations (created dedicated worker in supervisor configuration files, added the queue to the --queue= option in the config, etc.) but to no avail. I implemented report downloading through web sockets, and it works fine by itself, but when there are other things in the queue, the socket request gets handled after several items in the lower priority 'import' queue finish, leaving me to believe that I do not understand queue priorities very well. Is there a way to have a separate queue for socket requests that responds immediately to these requests?

UPDATE

I narrowed down the memory leak to the job that we were dispatching at the end of the import. However, I am not able to dispatch this job when attempting to import a very large file (30k+ rows). It errors out InvalidArgumentException: No handler registered for command [__PHP_Incomplete_Class] in Illuminate\Bus\Dispatcher (I think it's because it is running out of memory) and the php processes that were started by supervisor remain active and seem to consume the same amount of resources that they did at their peak (about 450 MB each). This keeps on going until I manually run php artisan queue:restart. For smaller files, the flow works as intendended: it runs the excel batches and at the end dispatches a job to the queue, which is also processed with success.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
prototorpedo
  • 55
  • 7
  • Does this help https://stackoverflow.com/questions/30121055/laravel-excel-massive-import . Similar issue: many rows and exel . cant you just increase the timeouts or memory limits on php. Id presume that you should have some errors in your php logs to help debugging the issue – Olli Tyynelä Sep 11 '17 at 13:01
  • Without seeing a mininum viable sample of how you are handling its really hard to give loads of tips or see that if you have some locical issues in your code. Is it possible that you are handling the same file multiple times. Eg some thing is causing a race issue with that large file – Olli Tyynelä Sep 11 '17 at 13:05
  • I'll try and post relevant code at some point. However, it doesn't seem to be the code, although it could use some optimizing. Excel processing goes through as expected, I can see the data correctly inserted in the database. For example, if I comment the dispatching of that other job, it works without any issue. The problem seems to be when dispatching that other job. For one reason or another, it throws the "No handler registered for command" error although I am implementing the SelfHandling interface and use the laravel bus dispatcher. – prototorpedo Sep 11 '17 at 13:12
  • Have you tried to increase the php timeouts / memorylimits made them unlimited? If the facts are that with small files it works but larger files fail. Or does the large file handle something different to cause the "No handler registered for command" to appear. – Olli Tyynelä Sep 11 '17 at 13:45
  • I increased the memory limit and it does seem to have some effect. The jobs fail half of the time now. I don't see how making the memory limit unlimited would be a good idea for the production server. – prototorpedo Sep 11 '17 at 15:24
  • If the app requires memory to function correctly you have two options: optimize or throw money on the server environment so it has more resources. The latter option might be the most cost efficient :). – Olli Tyynelä Sep 12 '17 at 08:45
  • But nevertheless id suggest on checking does it go through if you give it max available or unlimited. If it goes you can be sure that all the issues related to the memory or timeouts and make possible fixes based on that. – Olli Tyynelä Sep 12 '17 at 09:09

0 Answers0