2

I'm looking for some recommended configurations for IIS (7.5) logging for a reasonably high-traffic web-farm? Currently generating approx 500MB uncompressed logs on each of 4 web servers daily.

ODBC logging? To SQL Server seems like a lot of overhead for log files... Maybe to an open source database of some sort?

Should I just keep using log files and something like Analog to summarise them for points of interest?

What about Splunk? Could this be helpful?

Any recommendations or war-stories from people running IIS in larger environments and how you handle request logging?

Jason Duffett
  • 185
  • 1
  • 2
  • 8
  • My question is: What is your intended use for the logs (troubleshooting, traffic summary, page hits, etc)? The answer to that question should help you answer your original question. – joeqwerty Jun 03 '11 at 15:50

2 Answers2

1

I've been using AWStats for our large farms though I clean up the files first with LogParser (removing healthchecks, merging farm files (don't sort!), etc.). We don't have the size of files you do but we're definitely in the 300MB range and AWStats works fine. We don't need realtime so the processing happens on a jobs server at night and the stats are ready daily. We have stats for about 10 different sites. If you want to drill down to subsites it gets a bit messy but we've been using AWStats for about five or six years with no issues.

Would love to use Google Analytics but it's no good for intranet sites (security/privacy reasons).

Phillip Jubb
  • 141
  • 1
0

I've struggled with finding a good program to handle large log files. Somewhere around the size that you have is where most tools start to fail.

DeepMetrix LiveStats (no longer an active product) would fail around that log size.

SmarterStats (www.smartertools.com) is close, and you can give it a try. It will keep up on the log processing but it's slow viewing reports because of its own file system database.

Analog can handle a lot of logs and I believe works for this size although I haven't tried it in a few years.

Even Urchin struggles with large log files. It can probably handle your size, but it will eventually peak out too.

Log Parser is awesome as parsing large log files, but you'll need to build your own reports. I've reverted to some custom reporting on a couple really large sites.

Google Analytics is an option to consider too. It's a viable option for situations like yours.

Scott Forsyth
  • 16,449
  • 3
  • 37
  • 56