Laravel Flysystem & Rackspace CRON uploads fail with 401 while on-demand uploads work

Question

Our system generates various invoices every hour and uploads those to the cloud. Also it is possible to create the invoice on demand by clicking a button on our frontend.

When manually requesting to create the said invoice then it never fails to upload. As for the cron generated invoices after some time all uploads fail with:

Client error response
[status code] 401
[reason phrase] Unauthorized
[url] https://storage101.dfw1.clouddrive.com/v1/MossoCloudFS_930575/ <...> .pdf" @ Guzzle\Http\Exception\BadResponseException->factory

Which should mean that the token has expired which it probably is. Rackspace tokens last for 24 hours but the Laravel's Storage should automatically refresh the token.

Now here for some fun facts:

1) Every time our code is deployed with Capistrano the token seems to get refreshed and cron uploads work again for some time. Important thing to notice here is that every deploy creates a new folder similar to this /releases/201605190925 pulls the code, installs dependencies from scratch and if all goes well then symlinks this folder with the one Apache is showing.

2) Laravel jobs get handled on a different process than www-data

3) It's hard to track down if after deploy the upload works more than 24 hours or not. I suspect that sometimes it works more than that. But it's hard to track down since not every hour there are invoices that need to be generated. There are more developers who deploy than just me etc.

4) When cron fails and I get the failing notification I can immediately go and successfully generate the invoice. After that the cron still keeps failing. So it seems like these two instances have different tokens stored somewhere.

5) Cache cleaning with php artisan cache:clear doesn't seem to have any impact

6) Probably have tried restarting Apache service but without results

Since it has been going on for some time now I have tested various things and even contacted Rackspace at one point but they couldn't find anything odd from their end... Reminded me simply to catch the 401 error, update the token and try again. But Laravel and Flysystem should handle it somewhere by itself

Since nobody else seems to be having similar problems with Flysystem nor Laravel nor Rackspace I suspect that it's some kind of unique problem with the cron process. I was simply hoping that soon we have a refactored version of our system ready and the problem simply goes away. As for now it's still in development and might take another month.

Don't think it's related to code but anyway here's the upload line:

Storage::put($folder . '/' . $filename, file_get_contents($filePath));

Here's our config:

'default' => 'rackspace',

'disks' => [
    'rackspace' => [
        'driver'    => 'rackspace',
        'username'  => ' ... ',
        'key'       => ' ... ',
        'container' => ' ... ',
        'endpoint'  => 'https://identity.api.rackspacecloud.com/v2.0/',
        'region'    => 'DFW',
    ]
]

Any thoughts on the matter are appreciated.

score 3 · Accepted Answer · edited May 23 '17 at 12:00

Here I was once again trying to find a solution to the problem almost 2.5 months later. And I think it really clicked this time when I reread @Tommmm update.

So you restarted the worker queue and I started to think about the way I was running my supervisor/workers.

php artisan queue:work database --daemon --sleep=10 tries=3

--Daemon (as it was really nicely explained in this StackOverflow post by @the-shift-exchange )

In Laravel >=4.2 there was a --daemon command added. The way it works is simply keeps running the queues directly, rather than rebooting the entire framework after every queue is processed. This is an optional command that significantly reduces the memory and cpu requirements of your queue.

The important point with the --daemon command is that when you upgrade your application, you need to specifically restart your queue with queue:restart, otherwise you could potentially get all sorts of strange errors as your queue would still have the old code in memory.

Since the application never gets booted up again I feel that the token is also never renewed. As for the rest of the application still renews the token and uses the new one. That explains why all on-click actions were working and only the background tasks were failing. As to why deploying temporarily fixed our problem was that we had queue:restart in there after every deploy.

I am quite certain that this is the case and now there are two possibilities to start with. Either you restart your workers after some time (<24h for Rackspace) or you don't run the workers together with --daemon.

Edit:

Confirmed that adding following to Kernel.php restarts daemon workers after every 6 hours.

$schedule->command('queue:restart')->cron('0 */6 * * *')

score 2 · Answer 2 · edited May 23 '17 at 12:32

I've been having a similar issue. Documentation and error rates seem rather low for this problem. Reading here (https://github.com/rackspace/php-opencloud/issues/427) it looks like an expired token is the root cause.

The recommendation from the link above is to reauthenticate the client via

$client->authenticate();

But, the client is buried under a handful of commands. You should be able to access the underlying client via

Storage::disk('my-disk')->getDriver()->getAdapter()->getContainer()->getClient()->authenticate();

Unfortunately, I end up with a 35 curl error when trying that (SSL_CONNECT_ERROR).

There is a

hasExpired()

method available as well. If you encounter the same error, you should be able to combine this check with some sort of mechanism to restart the cron job or worker. While this is certainly far from an ideal fix, it should get you up and running. You would think this behavior would be caught and handled automatically.

5/25 Update: After spending another couple of hours trying to find a solution here, I threw in the towel and just created a cron job to restart the relevant workers every 12 hours. Supposedly the token is valid for 24 hours (according to HTTP 401 Fog::Storage::Rackspace::ServiceError), so the 12 hour window should safely keep the error from happening. While super hacky, it does keep that piece of the site up and running.

Thanks for the update. I'm currently also testing out various things with the problem. Our cron runs every hour and I tried combining `hasExpired` and `authenticate` without results. `hasExpires` seems to return false everytime. When I tried logging the token with cron (every minute) then I got different result every time. Not sure yet if relevant. `$client->getToken()` — Artur K., May 25 '16 at 12:21

Laravel Flysystem & Rackspace CRON uploads fail with 401 while on-demand uploads work

2 Answers2