Nginx Proxy to Files on Local Disk or S3

Question

So I'm moving my site away from Apache and onto Nginx, and I'm having trouble with this scenario:

User uploads a photo. This photo is resized, and then copied to S3. If there's suitable room on disk (or the file cannot be transferred to S3), a local version is kept.

I want requests for these images (such as http://www.mysite.com/p/1_1.jpg) to first look in the p/ directory. If no local file exists, I want to proxy the request out to S3 and render the image (but not redirect).

In Apache, I did this like so:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^p/([0-9]+_[0-9]+\.jpg)$ http://my_bucket.s3.amazonaws.com/$1 [P,L]

My attempt to replicate this behavior in Nginx is this:

location /p/ {
    if (-e $request_filename) {
        break;
    }
    proxy_pass http://my_bucket.s3.amazonaws.com/;
}

What happens is that every request attempts to hit Amazon S3, even if the file exists on disk (and if it doesn't exist on Amazon, I get errors.) If I remove the proxy_pass line, then requests for files on disk DO work.

Any ideas on how to fix this?

Answer 1

Shouldn't this be an example of using try_files?

location /p/ {
    try_files $uri @s3;
}

location @s3{ 
    proxy_pass http://my_bucket.s3.amazonaws.com;
}

Make sure there isn't a following slash on the S3 url

Answer 2

You could improve your s3 proxy config like this. Adapted from https://stackoverflow.com/a/44749584:

location /p/ {
    try_files $uri @s3;
}

location @s3 {
  set $s3_bucket        'your_bucket.s3.amazonaws.com';
  set $url_full         '$1';

  proxy_http_version     1.1;
  proxy_set_header       Host $s3_bucket;
  proxy_set_header       Authorization '';
  proxy_hide_header      x-amz-id-2;
  proxy_hide_header      x-amz-request-id;
  proxy_hide_header      x-amz-meta-server-side-encryption;
  proxy_hide_header      x-amz-server-side-encryption;
  proxy_hide_header      Set-Cookie;
  proxy_ignore_headers   Set-Cookie;
  proxy_intercept_errors on;

  resolver               8.8.4.4 8.8.8.8 valid=300s;
  resolver_timeout       10s;
  proxy_pass             http://$s3_bucket$url_full;
}

Answer 3

Thanks to keep my coderwall post :) For the caching purpose you can improve it a bit:

http {

  proxy_cache_path          /tmp/cache levels=1:2 keys_zone=S3_CACHE:10m inactive=24h max_size=500m;
  proxy_temp_path           /tmp/cache/temp;

  server {
    location ~* ^/cache/(.*) {
      proxy_buffering        on;
      proxy_hide_header      Set-Cookie;
      proxy_ignore_headers   Set-Cookie;
      ...
      proxy_cache            S3_CACHE;
      proxy_cache_valid      24h;
      proxy_pass             http://$s3_bucket/$url_full;
    }
  }

}

One more recommendation is to extend resolver cache upto 5 min:

resolver                  8.8.4.4 8.8.8.8 valid=300s;
resolver_timeout          10s;

Answer 4

break isn't doing quite what you expect nginx will do the last thing you ask of it, which makes sense if you start digging around making modules... but basically protect your proxy_pass with the does-not-exist version

if (-f $request_filename) {
    break;
}
if(!-f $request_filename)
    proxy_pass  http://s3;
}

Answer 5

I ended up solving this by checking to see if the file doesn't exist, and if so, rewriting that request. I then handle the re-written request and do the proxy_pass there, like so:

location /p/ {
  if (!-f $request_filename) {
    rewrite ^/p/(.*)$ /ps3/$1 last;
    break;
  }
}

location /ps3/ {
  proxy_pass http://my_bucket.s3.amazonaws.com/;
}