0

Recently a Rails 3 app we built and host had some issues with the Google Analytics tracker installed. This resulted in vastly diminished statistics being tracked during the last month. We have our production logs from the app and I'm wondering if anyone knows of any way to parse these to produce visitor statistics (similar to what web analytics packages would provide). We need to deliver a stats report this week and would like to have some account for the missing visitors. Any suggestions or help would be greatly appreciated!

Zachary Abresch
  • 756
  • 3
  • 8
  • 20

1 Answers1

1

Probably the better place to look would be your web server logs. 5 or 10 years ago all the popular analytics software gobbled up web server logs, and there are a few free ones our there. Google "web log analytics" and see if there's anything suitable.

The problem is, web logs contain all traffic, and for many websites, this can be from all sorts of sources you don't care about, like GoogleBot and others that crawl your site to add to search indexes ... and many more. Look for software that will try to filter these out, and will also know to ignore assets (JS, CSS, images, etc.). Analytics doesn't have to worry about this kind of stuff since it's based on cookies and javascript running in a real visitor's browser.

No matter how good these programs are, there are two things you'll need to take into account.

  1. Numbers will not align with GA, and you'll go crazy if you try to make them add up -- the differences can be astonishingly large, as much as 20% or more.
  2. It may be more work than it's worth to get the software configured -- even if you do, the level of detail pales in comparison to GA.

If you're handy with grep, the Rails log might help you get some quick-and-dirty counts (although they also record all traffic, unless users need to log in, in which case logs may be a little less noisy).

A different approach might be to look in your database -- is there anything you can track that acts as a proxy for a visit or any other goal you have been tracking? How useful this is depends entirely on your app and what you store in the database.

Some combination of the above may be the best way to get at something, but I hate to be the bearer of bad news -- it's very likely that what you're able to glean from logs creates more confusion than it's worth. Been there, tried that :-(

Tom

Tom Harrison
  • 13,533
  • 3
  • 49
  • 77
  • Thanks Tom. Unfortunately, we are using Passenger Phusion on this server and the Apache logs are filled with countless 'dummy connection' messages to 127.0.0.1. I'm sure there's a way to configure it to do something differently but that won't help much now. We do have some database numbers that can approximate important events and I feel very similarly about trying to correlate logs numbers vs. GA numbers. The account manager for this client is another story altogether. Thanks for your help and prompt response! – Zachary Abresch Nov 15 '12 at 14:35
  • Passenger shouldn't affect the normal apache logs, at least not in a typical setup -- maybe there's another log that's got the good stuff. Assuming Linux, check `/var/log/apache` or `/var/log/httpd` (or maybe `/var/www/logs`) to make sure. Just to count pageviews, the Rails log and some ninja greping might get you *something*. Actually a simple ruby script might be easier :-). – Tom Harrison Nov 15 '12 at 20:27