We have multiple tasks to run on the same api, since the api is rate limited we need to throttle how many calls are made (100 by 120 seconds) so we don't get locked out. Each lock lasts 30 minutes, would freeze the whole team and might create horrible issues.
System is built on Laravel, we are using queues to make this api calls. The first job we need is that once every night the system make around 950 calls to the api to sync whats on the api with our database. The second and third job would be on demand, it might need to run around 100 to 200 calls each whenever a user needs to. (multiple users might need this at the same time during the day, so the queue line should prevent us from surpassing the rate limit).
We set everything up in supervisor with only one worker, and it seemed to work fine. But then we realized it wasn't calling all the jobs on demand(2nd and 3rd kind), if a user needed to take action on 8 things, they would appear on the queue line but only run 2 or 3. The rest would get never pick up or don't run or even didn't fail. This got somewhat fixed by increasing the amount of workers to 8. After we did this, if the user needed to take action on 8 things worked, but our nightly 900 tasks began to fail.
Also we noticed that it worked with 8 jobs but if we let our users move up to 50 tasks, it also fails.
Our supervisor configuration on /etc/supervisor/conf.d:
[unix_http_server]
file = /tmp/supervisor.sock
chmod = 0777
chown= ubuntu:ubuntu
[supervisord]
logfile = /home/ubuntu/otmas/supervisord.log
logfile_maxbytes = 50MB
logfile_backups=10
loglevel = info
pidfile = /tmp/supervisord.pid
nodaemon = false
minfds = 1024
minprocs = 200
umask = 022
user = ubuntu
identifier = supervisor
directory = /tmp
nocleanup = true
childlogdir = /tmp
strip_ansi = false
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl = unix:///tmp/supervisor.sock
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /home/ubuntu/www/artisan horizon
autostart=true
autorestart=true
user=ubuntu
numprocs=8
redirect_stderr=true
stdout_logfile=/home/ubuntu/otmas/worker.log
Our queue config:
'connections' => [
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default',
'retry_after' => 460,
'block_for' => 140,
],
],
Our horizon config:
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'simple',
'processes' => 10,
'tries' => 3,
],
],
'local' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'simple',
'processes' => 3,
'tries' => 3,
],
],
'develop' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default','email'],
'balance' => 'auto',
'processes' => 3,
'tries' => 3,
'timeout' => 3,
'delay' => 3,
],
],
],
Our redis config on database.php:
'redis' => [
'client' => 'predis',
'default' => [
'host' => env('REDIS_HOST', '127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => env('REDIS_DB', 0),
'read_write_timeout' => -1,
],
'cache' => [
'host' => env('REDIS_HOST', '127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => env('REDIS_CACHE_DB', 1),
'read_write_timeout' => -1,
],
],
The nightly job:
class GetProjectTasks implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $project;
public $tries = 540;
public $timeout = 15;
public function __construct(ZohoProject $project)
{
$this->project = $project;
}
public function handle()
{
\Log::info("before redis get tasks for project: ".$this->project->id);
try {
$client = new GuzzleClient();
$accessToken = Setting::where('name','zoho_access_token')->get()->first();
$requestHeaders = [
'cache-control' => 'no-cache',
'Authorization' => 'Bearer '.$accessToken->value
];
$requestOptions = [
'headers' => $requestHeaders
];
$zohoTasksRestApi = Setting::where('name','zoho_projectapi_restapi')->get()->first()->value;
$project = $this->project->id;
} catch(Exception $e) {
Log::notice('Exception caught');
throw $e;
}
Redis::throttle('zohoprojecttasksget')->allow(85)->every(120)->then(function () use ($client, $zohoTasksRestApi, $portalId, $project, $requestOptions) {
try {
$zohoTasksResponse = $client->request('GET', $zohoTasksRestApi.'portal/'.$portalId.'/projects/'.$project.'/tasks/', $requestOptions);
$zohoTasksResult = json_decode((string) $zohoTasksResponse->getBody(), true);
if($zohoTasksResult) {
foreach ($zohoTasksResult['tasks'] as $key => $task) {
if (array_has($task, 'start_date') && array_has($task, 'end_date')) {
$taskStartDate = \Carbon\Carbon::createFromTimestamp((int)$task['start_date_long']/1000, 'America/Santiago');
$taskEndDate = \Carbon\Carbon::createFromTimestamp((int)$task['end_date_long']/1000, 'America/Santiago');
$zohoTask = ZohoTask::updateOrCreate(['id' => $task['id']], [
'name' => $task['name'],
'zoho_project_id' => $project,
'registry' => json_encode($task),
'start_date' => $taskStartDate,
'end_date' => $taskEndDate
]);
if(array_has($task, ['details.owners'])) {
if($task['details']['owners'][0]['name'] != 'Unassigned') {
foreach ($task['details']['owners'] as $key => $owner) {
$user = ZohoUser::find($owner['id']);
$zohoTask->assignees()->save($user);
}
}
}
} else {
$zohoTask = ZohoTask::updateOrCreate(['id' => $task['id']], [
'name' => $task['name'],
'zoho_project_id' => $project,
'registry' => json_encode($task),
]);
}
}
}
} catch(Exception $e) {
Log::notice('Exception caught');
throw $e;
}
}, function () use ($project) {
return $this->release(85);
});
}
}
Example of on demand job:
class ReplaceTask implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
protected $task, $taskToReplace;
public $tries = 140;
public $timeout = 15;
public function __construct(ZohoTask $task, ZohoTask $taskToReplace)
{
$this->task = $task;
$this->taskToReplace = $taskToReplace;
}
public function handle()
{
\Log::info("before redis replace task: ".$this->taskToReplace.' with task:: '.$this->task);
try {
$client = new GuzzleClient();
$accessToken = Setting::where('name','zoho_access_token')->get()->first();
$requestHeaders = [
'cache-control' => 'no-cache',
'Authorization' => 'Bearer '.$accessToken->value
];
$ZohoProjectsRestApi = Setting::where('name','zoho_projectapi_restapi')->get()->first()->value;
$task = $this->task;
$taskToReplace = $this->taskToReplace;
$project = $task->project;
} catch(Exception $e) {
Log::notice('Exception caught');
throw $e;
}
\Log::info("before redis task: ".$task->id.' to replace '.$taskToReplace->id.' Project: '.$project->id);
Redis::throttle('zohoupdatetask')->allow(50)->every(120)->then(function () use ($client, $ZohoProjectsRestApi, $portalId, $project, $requestHeaders, $task, $taskToReplace) {
try {
$assignee = $taskToReplace->assignees->last();
$assignResponse = $client->request('POST', $ZohoProjectsRestApi.'portal/'.$portalId.'/projects/'.$project->id.'/tasks/'.$task->id.'/', [
'headers' => $requestHeaders,
'form_params' => [
'start_date' => $taskToReplace->start_date->format('m-d-Y'),
'start_time' => $taskToReplace->start_date->format('h:i A'),
'person_responsible' => $assignee->id
]
]);
\Log::info($assignResponse->getBody());
if($assignResponse->getStatusCode() == 200) {
$task->assignees()->detach();
$assignResult = json_decode((string) $assignResponse->getBody(), true);
$taskRegistry = $assignResult['tasks'][0];
$taskStartDate = \Carbon\Carbon::createFromTimestamp((int)$taskRegistry['start_date_long']/1000, 'America/Santiago');
$taskEndDate = \Carbon\Carbon::createFromTimestamp((int)$taskRegistry['end_date_long']/1000, 'America/Santiago');
$task->start_date = $taskStartDate;
$task->end_date = $taskEndDate;
$task->registry = json_encode($taskRegistry);
$task->assignees()->save($assignee);
$task->save();
\Log::info("Task Ok move");
}
} catch(Exception $e) {
Log::notice('Exception caught');
throw $e;
}
}, function () use ($project, $task, $taskToReplace) {
\Log::info("Task hit throttle");
return $this->release(120);
});
\Log::info("after redis");
}
}
Horizon Tasks: (The ones on hold never get run or picked up or anything) Tasks appear on the queue, but never run or fail, it seems
