I have been having an issue in Laravel which I just can't seem to solve. As a part of our application, we are adding a lot of jobs to a queue and executing them using queue:work. This is setup with Supervisor and is working very well.
We are having trouble with just this one specific command (the rest are run without problems).
In our application, we are calling a custom shell script command using
Symfony\Component\Process\Process;
But I keep getting the "process exceeded the timeout of 60 seconds" error, no matter what.
Error: Symfony\Component\Process\Exception\ProcessTimedOutException: The process "'BIMTool.sh'" exceeded the timeout of 60 seconds. in /var/www/app/vendor/symfony/process/Process.php:1263
The BimTool.sh script iniates a series of commands involving Python & Anaconda3 to generate BIM-model data from IFC and point clouds. I even replaced the contents of BimTool.sh with "sleep 70" and I still got the error when run.
This is the code in question:
GenerateModelData.php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Support\Facades\Log;
use Symfony\Component\Process\Exception\ProcessFailedException;
use Symfony\Component\Process\Process;
use Carbon\Carbon;
use App\Models\MongoDB\VisualizerModel;
use App\Models\PointCloud;
use App\Models\Project;
class GenerateModelData implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $pointcloud;
/**
* The number of times the job may be attempted.
*
* @var int
*/
public $tries = 3;
/**
* The number of seconds the job can run before timing out.
*
* @var int
*/
public $timeout = 1200;
/**
* The number of seconds to wait before retrying the job.
*
* @var int
*/
public $backoff = 120;
/**
* Create a new job instance.
*
* @return void
*/
public function __construct(PointCloud $pointcloud)
{
$this->pointcloud = $pointcloud;
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
// These commands can take +8 hours to execute in some cases
ini_set('max_execution_time', 0);
set_time_limit(0);
$pointcloud = $this->pointcloud;
$modelID = $pointcloud->models[0]->model_id;
$model = VisualizerModel::find($modelID);
$IFCFileName = substr($model->filename, 0, -5);
$IFCFileLocation = "/var/www/app/storage/app/public/application/projects/{$model->project_uuid}/models/ifc/{$IFCFileName}.ifc";
$PTSFileLocation = "/var/www/app/storage/app/public/{$pointcloud->storage_location}";
$downSample = 15;
$uuid = $model->project_uuid;
$date = Carbon::parse($pointcloud->scan_date)->format('d/m/Y');
$modelDataID = $model->model_data_id;
$process = new Process(['BIMTool.sh',
$IFCFileLocation,
$PTSFileLocation,
$downSample,
$uuid,
$modelID,
$date,
$modelDataID
]);
$process->setTimeout(1200);
$process->setIdleTimeout(1200);
$process->start();
// Executes after the command finishes
if (!$process->isSuccessful()) {
throw new ProcessFailedException($process);
} else {
// Command was succesfull
// Some code here
}
}
/**
* Catch error if job fails.
*
* @return \Symfony\Component\Process\Exception\ProcessFailedException
*/
public function failed(ProcessFailedException $exception)
{
return $exception;
}
I have of course searched Stackoverflow (and the internet for that matter) for this problem already and have tried all the solutions I came across, but with no effect.
What I have tried so far (and none of them has worked)
- Changed setting max_input_time in php.ini to 0, -1 and 1200
- Changed setting max_execution_time in php.ini to 0, -1 and 1200
- $process->setTimeout(1200);
- $process->setIdleTimeout(1200);
- ini_set('max_execution_time', 0);
- set_time_limit(0);
- Added --timeout=0 to Supervisor command
Environment configuration
Linode running Ubuntu 18.04.5 LTS
LAMP-stack
Anaconda3
Laravel 5.8
Supervisor configuration
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work database --sleep=5 --tries=3 --timeout=0 --
queue=default
autostart=true
autorestart=true
user=bc
numprocs=1
redirect_stderr=true
stdout_logfile=/var/www/app/worker.log
What could be causing the process exceeded the timeout of 60 seconds error? I keep getting it no matter what I attempt.
EDIT: (this solved it for me)
So it turned out the queue has a "cache" as well. Running the command:
php artisan queue:restart
cleared the queue of cached versions of the jobs. What went wrong was that my changes to the code was never run, so make sure to run this command in order to make sure the job queue is using the most recent code.
Thx for the inputs!
