0

I have a Laravel application running in production, and there are a few APIs that used a lot. Something was creating a bottleneck and it used to stall our servers (3 with Load Balancer). After optimising the basics in Laravel, caching config, routes, data and so on, even resolving all n+1 issues, we were still having issues in peak times. Someone suggested we run strace on one of the nginx workers to see what is happening on the system level, so we did, and interesting enough, there are lots of redundant system calls where nginx tries to find files when APIs are called:

Part of trace:

240498 stat("/var/www/html/myProject/current/public/APIRequest/3d4f7518e04e9", 0x7ffc7ee6ff70) = -1 ENOENT (No such file or directory)
240498 stat("/var/www/html/myProject/current/public/APIRequest/3d4f7518e04e9", 0x7ffc7ee6ff70) = -1 ENOENT (No such file or directory)
240498 lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
240498 lstat("/var/www", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
240498 lstat("/var/www/html", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
240498 lstat("/var/www/html/myProject", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
240498 lstat("/var/www/html/myProject/current", {st_mode=S_IFLNK|0777, st_size=48, ...}) = 0
240498 readlink("/var/www/html/myProject/current", "/var/www/html/myProject/release"..., 4095) = 48
240498 lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
240498 lstat("/var/www", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
240498 lstat("/var/www/html", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
240498 lstat("/var/www/html/myProject", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
240498 lstat("/var/www/html/myProject/releases", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
240498 lstat("/var/www/html/myProject/releases/20201202085755", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0
240498 lstat("/var/www/html/myProject/releases/20201202085755/public", {st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0

Now the API is called with the ID 3d4f7518e04e9 in this case, and it tries to loop through the directories to find that file instead. But it is not a file, it is an API. We ran strace for less than 30 secs, and we have 5k such calls which don't make sense to me.

So, are these calls necessary? I don't think so, but tell me if I am wrong. And if I am right, how can I configure my nginx better so that these calls can be "caught in time" and resolved appropriately. Any ideas are welcome. :)

PS: We have tried apache with similar config also, same problem in strace appears.

Edit: Do I simply need some sort of a location directive in my site conf to solve this? I am using the basic nginx conf from the official docs https://laravel.com/docs/8.x/deployment#nginx with a few more additions like:

    location ~ \.php$ {
        fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        include fastcgi_params;

        fastcgi_buffer_size 128k;
        fastcgi_buffers 256 16k;
        fastcgi_busy_buffers_size 256k;
        fastcgi_temp_file_write_size 256k;
    }

    location ~ /\.(?!well-known).* {
        deny all;
    }

    client_body_buffer_size 10K;
    client_header_buffer_size 1k;
    client_max_body_size 8m;
    large_client_header_buffers 4 16k;

    client_body_timeout 12;
    client_header_timeout 12;
    keepalive_timeout 15;
    send_timeout 10;

EDIT:

location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

I do have try files as suggested by Danila Vershinin in an answer

Rohan
  • 135
  • 3
  • 12

1 Answers1

3

It looks like you are dealing with try_files performance issue. You can get rid of unnecessary stat system calls by eliminating try_files from your configuration.

The try_files directive provides a nice and simple boilerplate for creating SEO-friendly website.

However, the downside of this simplicity comes with an added cost of unnecessary stat system calls.

Since you know that, for e.g. all /api/ URLs must be routed through your PHP, there is no need to check any files for existence there and you can route through your Laravel bootstrap file unconditionally, e.g.:

location /api/ {
    fastcgi_param SCRIPT_FILENAME $document_root/index.php;
    include fastcgi_params;
    fastcgi_pass unix:/var/run/php-fpm/example.com.sock;
}

Moreover, in general, you want to have NGINX cache information about file/directories existence. This can be achieved through open_file_cache.

Danila Vershinin
  • 5,286
  • 5
  • 17
  • 21
  • Yes, I do have try_files as you mentioned, I also edited my question to add it there. So just a follow up question here, in my question, you see the common location block for php files, my question is, I should add another `location /api/` block with the same stuff as my regular php block right? That should work? – Rohan Dec 04 '20 at 07:50
  • Yes, 1) identify a URI pattern like `/api/` and create a prefixed NGINX location for it as in my answer 2) copy what you have for processing php files, e.g. from `location ~ \.php$` into the created location 3) ensure that line `fastcgi_param SCRIPT_FILENAME $document_root/index.php;` is included in the location. This will ensure that all requests to /api/ will route through your Laravel front controller. – Danila Vershinin Dec 04 '20 at 07:57
  • Got a lot of `FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream` errors when I added that block `location /APIRequest/`, maybe I am missing something, maybe I need to put it above the main php location block? – Rohan Dec 04 '20 at 08:23
  • Point 3 from my comment earlier was added? – Danila Vershinin Dec 04 '20 at 08:25
  • Oh sorry, I missed that, you mean I need to change `fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;` to `fastcgi_param SCRIPT_FILENAME $document_root/index.php;`? – Rohan Dec 04 '20 at 08:28
  • 1
    This worked, and my `lstat` calls in `strace` have come down from thousands every minute to just a few handful, thank you so much. – Rohan Dec 04 '20 at 09:01