9

I am currently running nginx/1.0.15 on a Centos 6.6 OS. The server has the following specs:

  • Intel(R) Atom(TM) CPU C2750 @ 2.40GHz (8 cores)
  • 32GB Ram
  • 5 x 6000 GB 7200 RPM (Raid 10)

The Problem

The server has a 1Gbit/s connection, however it tops out and bottlenecks after 400-500 mbit/s. Service starts to decline at roughly 100 connections.. and the speed with the server drops dramatically (despite having 50% bandwidth still available)

The NGINX server is strictly for serving static .mp4 files. Each file is typically 400-1200MB (700MB being the average)

I have tried many many configurations and just about all of them give me the same results.. I am extremely frustrated..

Server load also never passes 0.3.

Is there anything blatantly wrong or misguided in my configuration? Anything might help.

The Configurations

/etc/nginx/nginx.conf

user              nginx;
worker_processes  9;

error_log  /var/log/nginx/error.log;


pid        /var/run/nginx.pid;


events {
    worker_connections  51200;
    use epoll;
 }

worker_rlimit_nofile 600000;

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

#access_log  /var/log/nginx/access.log  main;
access_log off;

aio on;
sendfile        off;
tcp_nopush      off;
tcp_nodelay      on;

#keepalive_timeout  0;
keepalive_timeout  65;

output_buffers 1 3m;
#gzip  on;

include /etc/nginx/conf.d/*.conf;

open_file_cache          max=10000 inactive=5m;
open_file_cache_valid    2m;
open_file_cache_min_uses 1;
open_file_cache_errors   on;

}

/etc/nginx/conf.d/default.conf

server {
    listen       80 default_server sndbuf=32k;
    server_name  _;

    #charset koi8-r;

    #access_log  logs/host.access.log  main;

    include /etc/nginx/default.d/*.conf;


    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    location /Videos/ {
        root /home;
        gzip off;
        gzip_static off;

        mp4;
        mp4_max_buffer_size   300m;
    }

    location /stats {
        stub_status on;
    }

    error_page  404              /404.html;
    location = /404.html {
        root   /usr/share/nginx/html;
    }


    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}
masegaloeh
  • 18,236
  • 10
  • 57
  • 106
Kenneth
  • 131
  • 1
  • 1
  • 4

3 Answers3

6

The better start can be the set of following rules:

  1. disable logging and accept_mutex
  2. enable sendfile
  3. set sendfile_max_chunk

Configuration:

events {
    accept_mutex off;
}

access_log off;
sendfile on;
sendfile_max_chunk 512k;

New Nginx (1.7.11 or newer) feature thread pool can be really helpful in your case:

location / {
    root /home;
    aio threads;
    mp4;
}

On test samples it dramatically helps you to increase bandwidth from 1Gbps up to 9Gbps. Nine times! You have only 1Gbps but it makes it all utilised.

See more details: https://www.nginx.com/blog/thread-pools-boost-performance-9x/

Anatoly
  • 566
  • 3
  • 16
  • I have tried this today and unfortunely no improvements.. I did notice that my nginx worker processes seem to top out at 1918kb/s read spead, any idea what that limit may be? – Kenneth Jul 14 '15 at 14:21
  • @Kennysmoothx it less likely Nginx because you tried almost everything to configure it to serve large files efficiently – Anatoly Jul 15 '15 at 19:41
  • @Kennysmoothx this is ancient, but did you ever found a solution? I Would blame your hosting they might be overselling that's why you don't reach 1Gbps. Try a different hosting with same config see what you get. – Michael Rogers Jan 17 '19 at 18:30
5

A good first place to start is with the actual .mp4 files, where there is usually massive areas of improvement.

So before getting lost in tuning NGINX or Apache, first tune your .mp4 files.

For this post cinematic is like a movie or television show where each frame change is required. In other words, trying to retranscode a movie like "The Croods" to 1 fps (frame/second) would reduce quality to unwatchable.

And non-cinematic refers to screen captures like Webinars our courseware posted to Udemy.

First, consider the audio component of the file. If the audio component is primarily speaking, then use ffmpeg to retranscode the file where you copy the video stream (no change) + convert the stereo stream to mono. For many .mp4 files (non-cinematic) roughly 1/3 of movie file size is video + 1/3 is left audio channel + 1/3 is right audio channel. Changing from stereo to mono, can considerably reduce file size.

Second, retranscode audio using FDK-AAC (https://github.com/mstorsjo/fdk-aac) which produces far smaller files than other aac encoders. Most modern versions of ffmpeg automagickally build FDK-AAC these days. Even Macports now builds this. One consideration, for FDK to do it's real magic requires a stereo track + when using FDK stereo audio compresses far smaller than mono, so if you're using FDK, stick with stereo.

Third, for audio reduce bitrate. Many times this is 48k, so in general use -ar 44100 (ffmpeg) or for spoken (low fi) consider dropping to 22050.

Forth, set the frame rate of your video as low as possible. So if you're doing a screen capture, a frame may only change once in 10-60 seconds, so you can drop frame rate using -r $fps, many times from 30-60 fps to 1-5 fps + quality remains same while file size plummets.

Many times I compress non-cinematic files where every 1G reduces to 10-20M.

Fifth, make sure the faststart mov atom is at the front of your files, so your files can be streamed rather than downloaded.

My ffmpeg fdk parameters...

-c:a libfdk_aac -profile:a aac_he_v2 -afterburner 1 -signaling explicit_sbr -vbr 5 -ac 2 -ar 44100

In fact, here's a typical complete ffmpeg command...

The mp4 script is just a wrapper around ffmpeg that does things like take a guess at which audio + video tracks are in english (for multitrack avi + mkv files) + then build the ffmpeg command. What's of interest is the actual command, which is the residue of years of experiments.

Try running your files through ffmpeg extreme compression first, then see if the file weights are so low/small, there's no Web server tuning required.

Areas of experiment: -r $fps + -v:crf + -v:preset + -ar bitrate

A bit of experimenting will give you the settings for smallest file size + acceptable quality.

Many of the odd options like +genpts + clearing SAR/DAR are there to ensure .mp4 files play on Roku units. These are good to keep, in case you every setup your own Roku Channel, which is a free way to reach 5,000,000+ households.

My ffmpeg command...

imac> mp4 --dr --noisy foo.avi

tc: diag=v:!h264:mpeg4,a:!aac:ac3 title='Foo (TC)' Foo-640x480-veryfast-crf18-max-tc.mp4

cd '/Users/david/Downloads/Casper.A.Spirited.Beginning.1997.DVDrip.iNTERNAL.XviD-BPDcarrier' nice -19 ffmpeg -fflags +genpts -i "foo.avi" -map 0:0 -c:v libx264 -crf:v 18 -preset:v veryfast -tune:v film -level:v 4.1 -profile:v high -bufsize:v 5000k -vf setdar=dar=0,setsar=sar=0 -x264opts colorprim=bt709:transfer=bt709:colormatrix=bt709:fullrange=off -r 29.97 -movflags +faststart -map 0:1 -c:a libfdk_aac -profile:a aac_he_v2 -afterburner 1 -signaling explicit_sbr -vbr 5 -ac 2 -ar 44100 -metadata title='Foo (TC)' -threads 0 -f mp4 -benchmark Foo-640x480-veryfast-crf18-max-tc.mp4.tmp mv -f Foo-640x480-veryfast-crf18-max-tc.mp4.tmp Foo-640x480-veryfast-crf18-max-tc.mp4

David Favor
  • 171
  • 2
2

Turning on multi_accept worked for me (the video used to stop about halfway and the visitor was not able to listen/watch the other half, very frustrating).

The only things I have set in nginx.conf under the events is this:

events {
worker_connections 768;
multi_accept on;
}

**It works today LOL....tomorrow we'll just have to see if it stills play fully

Pierre.Vriens
  • 1,159
  • 34
  • 15
  • 19
kennyhendrick
  • 21
  • 1