I have a dedicated server with 1GB/s dedicated, 4GB ram and 4cpus. I have static files for download (from 300mb to 900mb). I was testing over Apache, Nginx and Lighttpd.
Apache makes too many threats and after 200 connections it goes very high so apache it's a NO GO...
Nginx after 100 connections it goes very high so it's a NO GO either.
Lighttpd so far is very good as is a single-threaded server. With 500 concurrent connections the load stays at 0.90 - 1.10 (very good) but I'm facing a download speed problem, it goes slower even when I have 1GBps dedicated port, I see the iptraf and with 500 concurrent connections it goes no more than 250000 KB/s. With apache and nginx sometimes it went to 700000 KB/s the upstream in the server. I switched between sendfile and writev in the config and it has the same result.
I'm not using any php or fast-cgi, just straight download directly to the file, for example: http://www.myserver.com/file.zip and it downloads the file.
I will attach some info here for you to help me figure it out.
Kernel 2.6
lighttpd.conf
# lighttpd configuration file
#
# use it as a base for lighttpd 1.0.0 and above
#
# $Id: lighttpd.conf,v 1.7 2004/11/03 22:26:05 weigon Exp $
############ Options you really have to take care of ####################
## modules to load
# at least mod_access and mod_accesslog should be loaded
# all other module should only be loaded if really neccesary
# - saves some time
# - saves memory
server.modules = (
# "mod_rewrite",
# "mod_redirect",
# "mod_alias",
"mod_access",
# "mod_cml",
# "mod_trigger_b4_dl",
# "mod_auth",
# "mod_status",
# "mod_setenv",
# "mod_proxy_core",
# "mod_proxy_backend_http",
# "mod_proxy_backend_fastcgi",
# "mod_proxy_backend_scgi",
# "mod_proxy_backend_ajp13",
# "mod_simple_vhost",
# "mod_evhost",
# "mod_userdir",
# "mod_cgi",
# "mod_compress",
# "mod_ssi",
# "mod_usertrack",
# "mod_expire",
# "mod_secdownload",
# "mod_rrdtool",
"mod_accesslog" )
## a static document-root, for virtual-hosting take look at the
## server.virtual-* options
server.document-root = "/usr/share/nginx/html/"
## where to send error-messages to
server.errorlog = "/www/logs/lighttpd.error.log"
# files to check for if .../ is requested
index-file.names = ( "index.php", "index.html",
"index.htm", "default.htm" )
## set the event-handler (read the performance section in the manual)
# server.event-handler = "freebsd-kqueue" # needed on OS X
server.event-handler = "linux-sysepoll"
#server.network-backend = "linux-sendfile"
server.network-backend = "writev"
# mimetype mapping
mimetype.assign = (
".pdf" => "application/pdf",
".sig" => "application/pgp-signature",
".spl" => "application/futuresplash",
".class" => "application/octet-stream",
".ps" => "application/postscript",
".torrent" => "application/x-bittorrent",
".dvi" => "application/x-dvi",
".gz" => "application/x-gzip",
".pac" => "application/x-ns-proxy-autoconfig",
".swf" => "application/x-shockwave-flash",
".tar.gz" => "application/x-tgz",
".tgz" => "application/x-tgz",
".tar" => "application/x-tar",
".zip" => "application/zip",
".mp3" => "audio/mpeg",
".m3u" => "audio/x-mpegurl",
".wma" => "audio/x-ms-wma",
".wax" => "audio/x-ms-wax",
".ogg" => "application/ogg",
".wav" => "audio/x-wav",
".gif" => "image/gif",
".jpg" => "image/jpeg",
".jpeg" => "image/jpeg",
".png" => "image/png",
".xbm" => "image/x-xbitmap",
".xpm" => "image/x-xpixmap",
".xwd" => "image/x-xwindowdump",
".css" => "text/css",
".html" => "text/html",
".htm" => "text/html",
".js" => "text/javascript",
".asc" => "text/plain",
".c" => "text/plain",
".cpp" => "text/plain",
".log" => "text/plain",
".conf" => "text/plain",
".text" => "text/plain",
".txt" => "text/plain",
".dtd" => "text/xml",
".xml" => "text/xml",
".mpeg" => "video/mpeg",
".mpg" => "video/mpeg",
".mov" => "video/quicktime",
".qt" => "video/quicktime",
".avi" => "video/x-msvideo",
".asf" => "video/x-ms-asf",
".asx" => "video/x-ms-asf",
".wmv" => "video/x-ms-wmv",
".bz2" => "application/x-bzip",
".tbz" => "application/x-bzip-compressed-tar",
".tar.bz2" => "application/x-bzip-compressed-tar"
)
# Use the "Content-Type" extended attribute to obtain mime type if possible
#mimetype.use-xattr = "enable"
## send a different Server: header
## be nice and keep it at lighttpd
# server.tag = "lighttpd"
#### accesslog module
accesslog.filename = "/www/logs/access.log"
## deny access the file-extensions
#
# ~ is for backupfiles from vi, emacs, joe, ...
# .inc is often used for code includes which should in general not be part
# of the document-root
url.access-deny = ( "~", ".inc" )
$HTTP["url"] =~ "\.pdf$" {
server.range-requests = "disable"
}
##
# which extensions should not be handle via static-file transfer
#
# .php, .pl, .fcgi are most often handled by mod_fastcgi or mod_cgi
static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
######### Options that are good to be but not neccesary to be changed #######
## bind to port (default: 80)
#server.port = 81
## bind to localhost (default: all interfaces)
#server.bind = "grisu.home.kneschke.de"
## error-handler for status 404
#server.error-handler-404 = "/error-handler.html"
#server.error-handler-404 = "/error-handler.php"
## to help the rc.scripts
#server.pid-file = "/var/run/lighttpd.pid"
###### virtual hosts
##
## If you want name-based virtual hosting add the next three settings and load
## mod_simple_vhost
##
## document-root =
## virtual-server-root + virtual-server-default-host + virtual-server-docroot
## or
## virtual-server-root + http-host + virtual-server-docroot
##
#simple-vhost.server-root = "/home/weigon/wwwroot/servers/"
#simple-vhost.default-host = "grisu.home.kneschke.de"
#simple-vhost.document-root = "/pages/"
##
## Format: <errorfile-prefix><status-code>.html
## -> ..../status-404.html for 'File not found'
#server.errorfile-prefix = "/home/weigon/projects/lighttpd/doc/status-"
## virtual directory listings
#dir-listing.activate = "enable"
## enable debugging
#debug.log-request-header = "enable"
#debug.log-response-header = "enable"
#debug.log-request-handling = "enable"
#debug.log-file-not-found = "enable"
#debug.log-condition-handling = "enable"
### only root can use these options
#
# chroot() to directory (default: no chroot() )
#server.chroot = "/"
## change uid to <uid> (default: don't care)
#server.username = "wwwrun"
## change uid to <uid> (default: don't care)
#server.groupname = "wwwrun"
#### compress module
#compress.cache-dir = "/tmp/lighttpd/cache/compress/"
#compress.filetype = ("text/plain", "text/html")
#### proxy module
## read proxy.txt for more info
#$HTTP["url"] =~ "\.php$" {
# proxy-core.balancer = "round-robin"
# proxy-core.allow-x-sendfile = "enable"
# proxy-core.protocol = "http"
# proxy-core.backends = ( "192.168.0.101:80" )
# proxy-core.max-pool-size = 16
#}
#### fastcgi module
## read fastcgi.txt for more info
## for PHP don't forget to set cgi.fix_pathinfo = 1 in the php.ini
#$HTTP["url"] =~ "\.php$" {
# proxy-core.balancer = "round-robin"
# proxy-core.allow-x-sendfile = "enable"
# proxy-core.check-local = "enable"
# proxy-core.protocol = "fastcgi"
# proxy-core.backends = ( "unix:/tmp/php-fastcgi.sock" )
# proxy-core.max-pool-size = 16
#}
#### CGI module
#cgi.assign = ( ".pl" => "/usr/bin/perl",
# ".cgi" => "/usr/bin/perl" )
#
#### SSL engine
#ssl.engine = "enable"
#ssl.pemfile = "server.pem"
#### status module
#status.status-url = "/server-status"
#status.config-url = "/server-config"
#### auth module
## read authentication.txt for more info
#auth.backend = "plain"
#auth.backend.plain.userfile = "lighttpd.user"
#auth.backend.plain.groupfile = "lighttpd.group"
#auth.backend.ldap.hostname = "localhost"
#auth.backend.ldap.base-dn = "dc=my-domain,dc=com"
#auth.backend.ldap.filter = "(uid=$)"
#auth.require = ( "/server-status" =>
# (
# "method" => "digest",
# "realm" => "download archiv",
# "require" => "user=jan"
# ),
# "/server-config" =>
# (
# "method" => "digest",
# "realm" => "download archiv",
# "require" => "valid-user"
# )
# )
#### url handling modules (rewrite, redirect, access)
#url.rewrite = ( "^/$" => "/server-status" )
#url.redirect = ( "^/wishlist/(.+)" => "http://www.123.org/$1" )
#### both rewrite/redirect support back reference to regex conditional using %n
#$HTTP["host"] =~ "^www\.(.*)" {
# url.redirect = ( "^/(.*)" => "http://%1/$1" )
#}
#
# define a pattern for the host url finding
# %% => % sign
# %0 => domain name + tld
# %1 => tld
# %2 => domain name without tld
# %3 => subdomain 1 name
# %4 => subdomain 2 name
#
#evhost.path-pattern = "/home/storage/dev/www/%3/htdocs/"
#### expire module
#expire.url = ( "/buggy/" => "access 2 hours", "/asdhas/" => "access plus 1 seconds 2 minutes")
#### ssi
#ssi.extension = ( ".shtml" )
#### rrdtool
#rrdtool.binary = "/usr/bin/rrdtool"
#rrdtool.db-name = "/var/www/lighttpd.rrd"
#### setenv
#setenv.add-request-header = ( "TRAV_ENV" => "mysql://user@host/db" )
#setenv.add-response-header = ( "X-Secret-Message" => "42" )
## for mod_trigger_b4_dl
# trigger-before-download.gdbm-filename = "/home/weigon/testbase/trigger.db"
# trigger-before-download.memcache-hosts = ( "127.0.0.1:11211" )
# trigger-before-download.trigger-url = "^/trigger/"
# trigger-before-download.download-url = "^/download/"
# trigger-before-download.deny-url = "http://127.0.0.1/index.html"
# trigger-before-download.trigger-timeout = 10
## for mod_cml
## don't forget to add index.cml to server.indexfiles
# cml.extension = ".cml"
# cml.memcache-hosts = ( "127.0.0.1:11211" )
#### variable usage:
## variable name without "." is auto prefixed by "var." and becomes "var.bar"
#bar = 1
#var.mystring = "foo"
## integer add
#bar += 1
## string concat, with integer cast as string, result: "www.foo1.com"
#server.name = "www." + mystring + var.bar + ".com"
## array merge
#index-file.names = (foo + ".php") + index-file.names
#index-file.names += (foo + ".php")
#### include
#include /etc/lighttpd/lighttpd-inc.conf
## same as above if you run: "lighttpd -f /etc/lighttpd/lighttpd.conf"
#include "lighttpd-inc.conf"
#### include_shell
#include_shell "echo var.a=1"
## the above is same as:
#var.a=1
sysctl.conf
# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding
net.ipv4.ip_forward = 0
# Controls source route verification
net.ipv4.conf.default.rp_filter = 1
# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0
# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1
# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1
# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536
# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536
# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736
# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296
# These ensure that TIME_WAIT ports either get reused or closed fast.
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1
# TCP memory
net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
# For Large File Hosting Servers
net.core.wmem_max = 1048576
#net.ipv4.tcp_wmem = 4096 87380 524288
net.ipv4.tcp_wmem = 4096 524288 16777216
Actual top command
top - 16:15:57 up 6 days, 19:30, 2 users, load average: 1.05, 0.85, 0.83
Tasks: 143 total, 1 running, 142 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.6%us, 2.8%sy, 0.0%ni, 64.7%id, 30.8%wa, 0.0%hi, 1.1%si, 0.0%st
Mem: 3914664k total, 3729404k used, 185260k free, 1676k buffers
Swap: 8388600k total, 9984k used, 8378616k free, 3340832k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28590 root 20 0 518m 75m 71m D 13.1 2.0 1:12.24 lighttpd
28660 root 20 0 15016 1104 812 R 1.9 0.0 0:00.02 top
1 root 20 0 19328 620 396 S 0.0 0.0 0:03.74 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.14 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.12 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:00.32 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0.0 0.0 0:01.96 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:00.19 watchdog/1
11 root RT 0 0 0 0 S 0.0 0.0 0:01.00 migration/2
12 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/2
13 root 20 0 0 0 0 S 0.0 0.0 5:04.44 ksoftirqd/2
14 root RT 0 0 0 0 S 0.0 0.0 0:00.23 watchdog/2
15 root RT 0 0 0 0 S 0.0 0.0 0:00.50 migration/3
16 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/3
17 root 20 0 0 0 0 S 0.0 0.0 0:01.84 ksoftirqd/3
18 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/3
iostat
Linux 2.6.32-220.7.1.el6.x86_64 (zlin) 05/01/2012 _x86_64_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.57 0.00 3.95 30.76 0.00 64.72
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 109.58 38551.74 149.33 22695425748 87908220
netstat -an |grep :80 |wc -l
259
iptraf
247270.0 kbits/sec
What should I change to make the clients download faster, they said sometimes it downloads slower than 10 KB/s
