How do I load a file using a URL from within a nginx module?

Question

I'm writing a nginx module which is expected to load load a remote file before sending a reply to the client.

The user passes an ID in the URL. I use that ID to load a URL to the remote file. My test looks something like this:

wget http://example.com/?id=123

The id 123 gets transformed to a URL such as

http://other.example.com/image/cute.png

Now I need to load cute.png from within my nginx module. Can I do that with an ngx_request or maybe an ngx_upstream? I've not been able to find any clear documentation that would show how to do that...

Update:

I've now (finally!) found the sub-request function:

ngx_int_t            rc;
ngx_str_t            uri;
ngx_http_request_t  *sr;

...

/* THIS IS WHAT WAS WRONG */
ngx_str_set(&uri, "http://other.example.com/image/cute.png");

rc = ngx_http_subrequest(r, &uri, NULL, &sr, NULL, 0);
if (rc != NGX_OK) {
    /* error */
}

However, instead of the answer from the 3rd party website, I'm getting 404 errors with the following HTML code:

<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

My feeling is that right now it queries my nginx server instead of using an external TCP connection to get the file from the 3rd party website...

Any idea whay could be wrong in such a simple statement?

score 0 · Answer 1 · answered Mar 15 '19 at 18:53

Okay, I found a solution for my case, although I do not really like it.

I changed the URL to not include the protocol, so instead of using:

http://other.example.com/image/cute.png

I will use:

/other.example.com/image/cute.png

and that works well with ngx_http_subrequest() when you also add a proxy_pass option like so:

location /other.example.com {
    proxy_pass http://other.example.com/;
}

So I see that as very problematic since you'll need to add one proxy_pass per 3rd party domain you access. That being said, many things are available to you such as caches for the proxied data. If you don't expect the files to be changed more than once a month, for example, you could cache those 3rd party files locally and it'd go much faster on the second and further accesses.

So it's possible to get remote files with nginx, it's just not possible to use just any URL. At least, from what I can see the proxy_pass does not allow for the domain name itself to be dynamically assigned.

How do I load a file using a URL from within a nginx module?

1 Answers1