3

What is the best way to monitor website traffic for a Google App Engine hosted website?

It's fairly trivial to put some code in each page handler to record each page request to the datastore, and now (thanks stackoverflow) I have the code to log the referring site.

There's another question on logging traffic using the datastore, but it doesn't consider other options (if there are any).

My concern is that the datastore is expensive. Is there another way? Do people typically implement traffic monitoring, or am I being over-zealous?

If I do implement traffic monitoring via the datastore, what fields are recommended to capture? What's good and/or common practise?

I'd go with: time-stamp; page; referer; IP address; username (if logged in). Any other suggestions?

Community
  • 1
  • 1
user1379351
  • 723
  • 1
  • 5
  • 18

4 Answers4

5

All of the items you mention are already logged by the built-in App Engine logger. Why do you need to duplicate that? You can download the logs at regular intervals for analysis if you need.

Daniel Roseman
  • 588,541
  • 66
  • 880
  • 895
3

People usually use Google Analytics (or something similar) as it does client-side tracking and gives more insight then server-side tracking.

If you only need server-side tracking then analysing logs should be enough. The problem with Log API is that it can be expensive because it does not do real querying: for every log search it goes thorough all logs (within range).

You might want to look at Mache, a tool that exports all GAE logs to Google BigQuery which has proper query functionality.

Another option would be to download logs and analyse them with a local tools. GAE logs are in Apache format so there are plenty of tools available.

Peter Knego
  • 79,991
  • 11
  • 123
  • 154
1

You can use the logging module and that comes with a separate quota limit.

7 MBytes spanning 69 days (1% of the Retention limit)

I don't know what the limit is but that's a line from my app so it seems to be quite large.

You can then add to the log with 

logging.debug("something to store")

if it does not already come with what you need, then read it out locally with:

appcfg.py --num_days=0 request_logs appname/ output.txt
Paul Collingwood
  • 9,053
  • 3
  • 23
  • 36
0

Anything you write out via System.err.println (or the python equivalent) will automatically be appended to the app engine log. So, for example, you can create you own logging format, put println's on all your pages, and then download the log and grep for that format. So for example, if this is your format:

MYLOG:url:userid:urlparams

then download the log and pipe it through grep ^MYLOG and it would give you all the traffic for your site.

eeeeaaii
  • 3,372
  • 5
  • 30
  • 36