different behaviour per filename and backend routing in NGINX

Question

It's few days i'm bouncing my head on this problem but i cannot find a "elegant" solution. I would like to use a NGINX server as streaming origin for some live channel to be distributed trough different CDNs.

At a first sight the problem is simple, but when it come to the real-life scenario starts becoming complex and i want to know your opinions....

To begin, i have multiple transcoders configured as backends, and NGINX has to be able to route to the right encoder. To achieve this i used something like:

upstream transcoder_pool_01 {
 server 192.168.1.1;
 server 192.168.1.2;
}

upstream transcoder_pool_02 {
 server 192.168.1.3;
 server 192.168.1.4;
}

map $livechannel $backend {
 channel1 transcoder_pool_01;
 channel2 transcoder_pool_02;
 channel3 transcoder_pool_03;
}

And up to here everything fine. The difficul part comes now.

My transcoder/packagers generate streams with the following scheme:

Master HLS manifest: http://ip_address/liveapplication/channel_name/master.m3u8

Single HLS bitrate manifests http://ip_address/liveapplication/channel_name/channel_name-RESOLUTION/playlist.m3u8

DASH manifest: http://ip_address/liveapplication/channel_name/manifest.m3u8

Segments: http://ip_address/liveapplication/channel_name/channel_name-RESOLUTION/chunk-xxxxxxx.ts (or chunk-xxxxxx.mp4, or chunk-xxxxxx.m4s)

please note that the packagers share the same liveapplication and segments for both DASH and HLS.

Now, for reasons out of my control, the CDN needs DASH and HLS to be handled with different paths. And in addition to this the same Origin stream is re-used multiple times in different enviromments and channel-blocks. Due to this reason, from the CDN/Edges my Origin receives requests like these:

http://originserver/live-hls/environment-01/channel-block01/channel-name/master.m3u8

or

http://originserver/live-dash/environment-01/channel-block02/channel-name/manifest.mpd

or

http://originserver/live-dash/environment-02/channel-block01/channel-name/channel-name-SD/playlist.m3u8

or

http://originserver/live-dash/environment-02/channel-block01/channel-name/channel-name-SD/chunk-002314.m4s

and so on....

At this point, before forwarding the requests what i need is:

change "live-dash" or "live-hls" to a common "liveapplication"
strip out from the URI the environment and the channel-block
pass the remaining part as it is but: 3a) setting a very long cache for files called master.m3u8 and manifest.mpd, and for all the media chunks 3b) setting a very short cache for the files called playlist.m3u8 (note that those files are located in the same subdirectory in which the relative chunks are)

I was thinking to temporary solve the problem defining a server similar to:

server {
 listen 80;
 server_name originserver;
 proxy_redirect off;
 proxy_http_version 1.1;
 proxy_read_timeout 10s;
 proxy_send_timeout 10s;
 proxy_connect_timeout 10s;
 proxy_ignore_headers Cache-Control;
 proxy_ignore_headers Set-Cookie;
 proxy_cache liveorigin_cache;
 proxy_cache_methods GET HEAD;
 proxy_cache_key $uri;
 proxy_cache_lock on;
 proxy_cache_lock_age 5s;
 proxy_cache_lock_timeout 1h;
 # cache valid for 5 seconds where not specified differently
 proxy_cache_valid 200 302 5s;
 proxy_cache_valid 404 1s;

location ~ /(live-hls|live-dash)/([a-z0-9_\.-]+)/([a-z0-9_\.-]+)/([a-z0-9_\.-]+)/(master.m3u8|manifest.mpd) {
     proxy_cache_valid 200 400s;
     add_header "X-MasterManifest-Tag" "true";
     proxy_pass http://$backend/liveapplication/$4/$5;
}

location ~ /(live-hls|live-dash)/([a-z0-9_\.-]+)/([a-z0-9_\.-]+)/([a-z0-9_\.-]+)/(.*)/playlist.m3u8 {
     proxy_cache_valid 200 2s;
     add_header "X-Playlist-Tag" "true";
     proxy_pass http://$backend/liveapplication/$4/$5/playlist.m3u8;
}

location ~ /(live-hls|live-dash)/([a-z0-9_\.-]+)/([a-z0-9_\.-]+)/([a-z0-9_\.-]+)/(.*)/(.*).(ts|mp4|fmp4|m4s) {
     proxy_cache_valid 200 2s;
     add_header "X-Segments-Tag" "true";
     proxy_pass http://$backend/liveapplication/$4/$5/$6/$7;
}

p.s.: the added headers are for me to verify that the location is actually matched as expected....

But it definitely doesn't work.

I temporarly "patched" the problem replacing all the locations with a single location like:

       location ~ /(live-hls|live-dash)/([a-z0-9_\.-]+)/([a-z0-9_\.-]+)/([a-z0-9_\.-]+)/(.*) {
                set $livechannel $4;
                add_header "X-General-Tag" "true";
                proxy_pass http://$backend/live/$4/$5;
       }

That works but of course all the files are cached in the same way, and considered that there is a indefinite number of edges contacting this origin i really need to reduce the load on the transcoder.

And, i also believe there is an approach a lot better than the one I thought to use.

Maybe using nested locations? I'm really lost.

Thanks for your help. Alex

different behaviour per filename and backend routing in NGINX

0 Answers0