awstats server in dynamic infrastructure

Question

I have a linux server dedicated to awstats. This server hosts awstats application and generate/publish statistics for several websites hosted by several webservers deployed in my infrastructure.

In order to do this job at this moment each webserver exports via NFS its web-access-logs directories and the awstats server mounts them in read only. In this way everything works fine and awstats can read all servers logs.

The main problem of this solution is that whenever i turn off/on a webserver i have to reconfigure both the NFS mounts and the Awstats configuration inside awstats server. Considering that i have a dynamic environment (private cloud) it happens that i have to turn off and on webservers many times during the day based on the load i have on them.

Do you know any other possible and smart configuration i can apply to the awstats in order to accomplish this need without having to reconfigure it every time?

Thank you very much.

Answer 1

When I do similar things in AWS or similar clouds, I use autofs to mount the log directories. That means that the log directories are mounted on a demand-driven basis, ie not until needed and unmounted shortly thereafter. It also means that access to a directory that maps to a down server involves a short wait before an empty directory is returned, instead of a long NFS hang.

Then I maintain a single, central list of the currently live servers to poll, so that the log job doesn't poll servers which are down, and doesn't even have the autofs wait; but that is a mere refinement.

Answer 2

An alternative would be to configure the webservers to transmit their access logs by means of syslog to your awstats server.

That would mean zero configuration change to the awstats configuration when you bring webservers up and down.

On an Apache webserver you might now have something like:

ErrorLog "/var/log/www/error.log"
CustomLog "/var/log/www/access.log" combined

Replace those with a pipe to logger (a standard utility which converts stdin to syslog messages) and optionally use tee to duplicate the log messages so the existing log file will also be maintained:

ErrorLog  "|/bin/sh -c '/usr/bin/tee -a /var/log/www/error.log  | /usr/bin/logger -thttpd -plocal6'"
CustomLog "|/usr/bin/logger -thttpd -plocal7" combined

And then instruct the syslog daemon on each webserver to send log files to your awstats server:

# syslog.conf
local6.* @192.168.0.1    
local7.* @192.168.0.1

Where the incoming messages can be collected and stored for further processing by awstats.

If you have a number of different virtual hosts, you might want to add an additional LogFormat directive that also records the VirtualHost:

 LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined-with-vhost

and use

CustomLog "|/usr/bin/logger -thttpd -plocal7" combined-with-vhost

You can then do some pre-processing and split records for each VirtualHost to it's own log file, or change your awstats configuration to deal with such records with %virtualname and create correct statistics.