How to make Analog to parse only one week of logs?

Question

On my windows server, I use analog to parse my log files and analyze them. The logs are daily and named as exYYMMDD.log How i can set in the analog cfg file to parse only one week of logs?

Answer 1

I just started using analog and reportmagic myself..

I grabbed this out of the howto in the docs:

You can also include the date in the LOGFILE name, by using the following codes.

%D date of month
%m month name, in English
%M month number
%y two-digit year
%Y four-digit year
%H hour
%n minute
%w day of week, in English

So for example,
LOGFILE access_log%Y%M.log

will look for the logfile access_log200109.log, if it's September 2001. The date used is actually the TO date if one was specified, and otherwise the time of the start of the program. So for example, you can look at all of last month's logfiles with the commands

Which should give you a little help in setting up exactly what logfiles you wish to parse. In addition there's this corresponding entry, that details the "TO" and "FROM" dates:

There is also one other pair of commands which belongs in this category, namely the FROM > and TO commands. These specify a time period to restrict the analysis to. The simplest usage of these commands is FROM yyMMdd or FROM yyMMdd:hhmm, where yy represents the last > two digits of the year (analog assumes that the year is between 1970 and 2069), MM represents the month, dd is the date, hh the hour, and mm the minute. So, for example, to > analyse only requests from 1st July 1999 to 1pm on 15th June 2000 I would use the configuration

FROM 990701
TO 000615:1300

Alternatively, each of the components can be preceded by + or - to represent time relative to the time at which the program was invoked. In this case, the date can have more than 2 digits. This allows constructions like

FROM -01-00+01 # from tomorrow last year
TO -00-0131 # to the end of last month (OK even if last month didn't have 31 days)
FROM -00-00-112
TO -00-00-01 # statistics for the last 16 weeks
FROM -00-00-00:-06+01 # statistics for the last 6 hours

There are command line abbreviations +F and +T for the FROM and TO commands; for example, +T-00-00-01:1800 looks at statistics until 6pm yesterday. -F and -T turn off the from and to, as do FROM OFF and TO OFF.

With those, you can likely derive what you need. I'd hazard an educated guess that your answer lays within the "FROM" and "TO" .cfg entries.

As one more alternative, you could always skip those above and simply write a batch script that will move/archive or delete the previous week of logfiles -out_ of the default "\LogFiles\W3SVC1" location to say, something like "D:\IIS-Logfiles-Archive", and then run the analog analyzer for the day. Some pseudocode that might help you get started and moving in the right direction would be:

(and again, this is PSEUDO-code. Rough, ugly and fits like a wet wool sweater. There probably isn't a need to delete the directory, but I'm cribbing from another script that I use, mmmkay? :)

net stop all_your_webservices_here
cd C:\WINDOWS\system32\LogFiles\W3SVC1
del /f /q .
cd c:\
rd /s /q C:\WINDOWS\system32\LogFiles\W3SVC1
mkdir C:\WINDOWS\system32\LogFiles\W3SVC1
net start all_your_webservices_here
"C:\Program Files\analog 6.0\analog.exe" && rmagic

Sorry, that last little bit in there is if you've got reportmagic installed and running as well. If you don't, I HIGHLY recomend it. Makes analog pretty, impresses management. Mmm, Mmmm good. Like soup or something. :)

That being said, I'm going to guess that you're wanting to restrict analyzing to a week because it's taking forever for analog to parse your logs. (This is a totally random guess, and colored by my opinion, because I sorta/kinda saw the same thing when initially setting it up :)

Be advised, once you parse a set of logfiles, (say a few days worth, or a week's worth) it may take a while, however re-running analog to examine any following logfiles is extremely quick. To summarize; create a scheduled task for analog to run once a day and it while the initial run will take some time, each successive run will be much faster than the first.

HTH, mate...

Answer 2

Dude, I just thought of this after posting. Maybe this will help a little more, since most of the links to, "Here's how Bob did his config, and here's how Jerry did his here, which is nice and all, etc." are no longer valid.

In the hopes that it can help you out a little more, here's my analog.cfg file:

"#" The below WAS production, and became a test.
"#"
"#" LOGFILE C:\WINDOWS\system32\LogFiles\W3SVC1\ex%y%M%D.*
"#"
"#" It worked, but only gave us one logfile at a time.
"#"
"#" The below is a test.
"#"
"#" LOGFILE C:\WINDOWS\system32\LogFiles\W3SVC1*.log
"#"
"#" The above worked, because we want to parse ALL the logfiles,
"#" sequentially. Sticking with this for now.
"#"

"#"
"#" This line records how long it took to generate reports.
"#"

RUNTIME ON

"#"
"#" This line re-sets the logfile to localtime. Took a while to
"#" figure out as well.
"#"

LOGTIMEOFFSET -360
LOGFILE C:\WINDOWS\system32\LogFiles\W3SVC1*.log

"#"
"#" This line tells "analog" to dump it's output into "computer readable"
"#" format, for reportmagic to use. If we didn't use this, we would
"#" tell it to output it in html format for human, or browser-readable
"#" format.
"#"

OUTPUT COMPUTER

"#"
"#" This line tells analog specifically WHERE to dump it's output.
"#"

OUTFILE D:\inetpub\wwwroot\stats\report.dat

"#"
"#" These lines tell analog where the charts go.
"#"

LOCALCHARTDIR "D:\inetpub\wwwroot\stats\charts\%Y.%M."
CHARTDIR "charts/%Y.%M."

"#"
"#" These lines set the pretty text in the html
"#"

HOSTNAME "[My company's intranet]"
HOSTURL http://mycompany_intranet

"#"
"#" These lines tell analog how to resolve it's hostnames. We
"#" need to work on these some, because the "dnscache.txt" file
"#" simply continues to grow as this runs and results in us having
"#" to exert some maintenance. I was unable to get it to resolve
"#" DIRECTLY from the network, however I'm sure that it's possible
. "#"

DNS WRITE
DNSFILE dnscache.txt
DNSLOCKFILE dnslock.txt
DNSGOODHOURS 8

"#"
"#" Here's some pre-set, pre-module-load options that we need to set in the very beginning.
"#" They're kinda like the ones above, but you can consider them meta-tags or meta-variables.
"#"

REQLINKINCLUDE pages
REFLINKINCLUDE *
REDIRREFLINKINCLUDE *
FAILREFLINKINCLUDE *
SUBBROW /
SUBTYPE .gz,.Z

"#" Add whichever of these types of pages you have on your server, or others.

PAGEINCLUDE *.shtml
PAGEINCLUDE *.html
PAGEINCLUDE *.asp
PAGEINCLUDE *.jsp
PAGEINCLUDE *.js
PAGEINCLUDE *.cfm
PAGEINCLUDE *.css
PAGEINCLUDE *.pl
PAGEINCLUDE *.php
PAGEINCLUDE *.swf

TYPEALIAS .html ".html [Hypertext Markup Language]"
TYPEALIAS .htm ".htm [Hypertext Markup Language]"
TYPEALIAS .shtml ".shtml [Server-parsed HTML]"
TYPEALIAS .ps ".ps [PostScript]"
TYPEALIAS .gz ".gz [Gzip compressed files]"
TYPEALIAS .tar.gz ".tar.gz [Compressed archives]"
TYPEALIAS .jpg ".jpg [JPEG graphics]"
TYPEALIAS .jpeg ".jpeg [JPEG graphics]"
TYPEALIAS .gif ".gif [GIF graphics]"
TYPEALIAS .png ".png [PNG graphics]"
TYPEALIAS .txt ".txt [Plain text]"
TYPEALIAS .cgi ".cgi [CGI scripts]"
TYPEALIAS .pl ".pl [Perl scripts]"
TYPEALIAS .css ".css [Cascading Style Sheets]"
TYPEALIAS .class ".class [Java class files]"
TYPEALIAS .pdf ".pdf [Adobe Portable Document Format]"
TYPEALIAS .zip ".zip [Zip archives]"
TYPEALIAS .hqx ".hqx [Macintosh BinHex files]"
TYPEALIAS .exe ".exe [Executables]"
TYPEALIAS .wav ".wav [WAV sound files]"
TYPEALIAS .avi ".avi [AVI movies]"
TYPEALIAS .arc ".arc [Compressed archives]"
TYPEALIAS .mid ".mid [MIDI sound files]"
TYPEALIAS .mp3 ".mp3 [MP3 sound files]"
TYPEALIAS .doc ".doc [Microsoft Word document]"
TYPEALIAS .rtf ".rtf [Rich Text Format]"
TYPEALIAS .mov ".mov [Quick Time movie]"
TYPEALIAS .mpg ".mpg [MPEG movie]"
TYPEALIAS .mpeg ".mpeg [MPEG movie]"
TYPEALIAS .asp ".asp [Active Server Pages]"
TYPEALIAS .jsp ".jsp [Java Server Pages]"
TYPEALIAS .cfm ".cfm [Cold Fusion]"
TYPEALIAS .php ".php [PHP]"
TYPEALIAS .js ".js [JavaScript code]"

"#" x=General
"#" 1=Yearly Q=Quarterly m=Monthly W=Weekly D=Daily d=DailySum
"#" H=Hourly h=HourlySum w=HoursOfWeek 4=15minly 6=15minlySum 5=5minly
"#" 7=5minlySum
"#" o=Domain Z=Organisat. S=Host s=RefSite f=ReferrerURL
"#" n=SearchWord N=SearchQuery B=Browser b=BrowserSum p=OS
"#" i=Dir. r=File t=FileType z=FileSize c=Status
"#" I=Failure L=FailHost K=FailRef E=Redir l=RedirHost k=RedirRef
"#" v=VHost R=RedirVHost M=FailVHost u=User j=RedirUser J=FailUser
"#" y=IntSearchWord Y=IntSearchQuery P=ProcTime

WEEKBEGINSON MONDAY
MARKCHAR +
REPSEPCHAR none
COMPSEP " "
RAWBYTES OFF
BYTESDP 2
NOROBOTS ON
SEARCHCHARCONVERT ON
BARSTYLE i
PNGIMAGES ON
HTMLPAGEWIDTH 65
ASCIIPAGEWIDTH 76
MINGRAPHWIDTH 15
ALLCHART ON
ALLGRAPH P


"#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#"
"#" SERVICE REPORTS - HOW WELL WE ARE DELIVERING THE PAGES "#"
"#" "#"
"#" Status - whether the pages were delivered okay, or failed "#"
"#" Proctime - time to dispatch files "#"
"#" Failure - people requests these pages, and it failed "#"
"#" Failhost, Failref - failures arose from links on these hosts, on these URLs "#"
"#" Redir - people requested these files but were redirected "#"
"#" Redirhost, redirref - redirections arose from links on these hosts and URLs "#"
"#" "#"
"#" COLS, CHART, SORTBY, FLOOR, SUB, ARGS, LINKINCLUDE - as above "#"
"#" 304ISSUCCESS - 304 Not Modified means user saw a cached copy: i.e. success! "#"
"#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#""#"

REPORTORDER x1QmWDdHwh4657oZSlLujJkKfsNnBbpvRMcPztiEIYyr

DOMAIN ON
DOMCHART ON
DOMCOLS PQ
DOMSORTBY PAGES
DOMFLOOR -500p
SUBDOMAIN ...
SUBDOMSORTBY PAGES
SUBDOMFLOOR 1.5%p

DIRECTORY ON
DIRCHART ON
DIRCOLS PQc
"#"
"#" Excluding all "local" and "images" directories, as per web team
"#" request.
"#"
DIREXCLUDE */local
DIREXCLUDE /images
"#"
DIRSORTBY PAGES7
SUBDIRSORTBY PAGES7
DIRFLOOR 0q
SUBDIRFLOOR 0q
SUBDIR /////

FILETYPE ON
TYPECHART ON
TYPECOLS Rb
TYPESORTBY BYTES
SUBTYPE .gz,.Z

SIZE ON
SIZECHART ON
SIZECOLS ScR

ORGANISATION ON
ORGCHART ON
ORGCOLS Pb
ORGSORTBY PAGES
ORGFLOOR -500p
SUBORGSORTBY PAGES
SUBORG ....com
SUBORGFLOOR 25.0%p

HOST ON
HOSTCHART ON
HOSTCOLS Pb
HOSTSORTBY PAGES
HOSTFLOOR -500p

BROWSERREP ON
BROWREPCHART ON
BROWREPCOLS Rb
BROWREPSORTBY REQUESTS
BROWREPFLOOR -400r

BROWSERSUM ON
BROWSUMCHART ON
BROWSUMCOLS PQq
BROWSUMSORTBY PAGES7
BROWSUMFLOOR -200q
SUBBROW ///
SUBBROWSORTBY PAGES7
SUBBROWFLOOR -400q

STATUS ON
STATUSCHART ON
STATUSCOLS R
STATUSSORTBY ALPHABETICAL
STATUSFLOOR -200r
304ISSUCCESS ON

OSREP ON
OSCHART ON
OSCHARTEXPAND Windows
OSCOLS PQq
OSSORTBY PAGES7
OSFLOOR -200q
SUBOSSORTBY PAGES7
SUBOSFLOOR -500q

PROCTIME ON
PROCTIMECOLS R
PROCTIMECHART ON

USER ON
USERCOLS Rb
USERCHART ON
USERSORTBY REQUESTS
USERFLOOR -500r

REQUEST ON
REQCHART ON
REQCOLS RSc
REQSORTBY REQUESTS7
REQFLOOR 0.5%s
REQARGSSORTBY REQUESTS7
REQARGSFLOOR 0.5%s
REQEXCLUDE *
REQINCLUDE *.html
REQINCLUDE *.htm
REQINCLUDE *.zip
REQINCLUDE *.gz
REQINCLUDE *.Z
REQINCLUDE .pdf
REQLINKEXCLUDE /cgi-bin/
REQLINKEXCLUDE /scripts/*
REQLINKINCLUDE *
REQLINKINCLUDE *.zip
REQLINKINCLUDE *.pdf

FAILURE ON
FAILCHART ON
FAILCOLS RS
FAILSORTBY REQUESTS7
FAILFLOOR -300s
FAILARGSSORTBY REQUESTS7
FAILARGSFLOOR 10s
FAILLINKEXCLUDE *

FAILHOST ON
FAILHOSTCHART ON
FAILHOSTCOLS RS
FAILHOSTSORTBY REQUESTS7
FAILHOSTFLOOR -500s

FAILUSER ON
FAILUSERCOLS R
FAILUSERCHART ON
FAILUSERSORTBY REQUESTS
FAILUSERFLOOR -400r

FAILREF ON
FAILREFCHART ON
FAILREFCOLS RS
FAILREFSORTBY REQUESTS7
FAILREFFLOOR -300s
FAILREFARGSSORTBY REQUESTS7
FAILREFARGSFLOOR 10s
FAILREFLINKINCLUDE *

REDIR ON
REDIRCHART ON
REDIRCOLS SR
REDIRSORTBY REQUESTS7
REDIRFLOOR -300s
REDIRARGSSORTBY REQUESTS7
REDIRARGSFLOOR 500s

REDIRHOST ON
REDIRHOSTCHART ON
REDIRHOSTCOLS SR
REDIRHOSTSORTBY REQUESTS7
REDIRHOSTFLOOR -500s

REDIRREF ON
REDIRREFCHART ON
REDIRREFCOLS RS
REDIRREFSORTBY REQUESTS7
REDIRREFFLOOR -300s
REDIRREFARGSSORTBY REQUESTS7
REDIRREFARGSFLOOR 500s
REDIRREFLINKINCLUDE *

REFSITE ON
REFSITECHART ON
REFSITECOLS PQ
REFSITESORTBY PAGES7
REFSITEFLOOR -500q
REFDIRSORTBY PAGES7
REFDIRFLOOR 50.5%q

REFERRER ON
REFCHART ON
REFCOLS PQ
REFSORTBY PAGES7
REFLINKINCLUDE *
REFFLOOR -500q
REFARGSSORTBY PAGES7
REFARGSFLOOR 50.5%q

VHOST ON
VHOSTCHART ON
VHOSTCOLS Rb
VHOSTFLOOR -200b
VHOSTSORTBY BYTES

REDIRVHOST ON
REDIRVHOSTCHART ON
REDIRVHOSTCOLS R
REDIRVHOSTFLOOR -200r
REDIRVHOSTSORTBY REQUESTS

REDIRUSER ON
REDIRUSERCOLS R
REDIRUSERCHART ON
REDIRUSERSORTBY REQUESTS
REDIRUSERFLOOR -200r

SEARCHWORD ON
SEARCHWORDCHART ON
SEARCHWORDCOLS PQq
SEARCHWORDSORTBY PAGES7
SEARCHWORDFLOOR 1.5%q

SEARCHQUERY ON
SEARCHQUERYCHART ON
SEARCHQUERYCOLS Q
SEARCHQUERYSORTBY PAGES7
SEARCHQUERYFLOOR -500q

INTSEARCHQUERY ON
INTSEARCHQUERYCOLS R
INTSEARCHQUERYCHART ON
INTSEARCHQUERYFLOOR -300r
INTSEARCHQUERYSORTBY REQUESTS

INTSEARCHWORD ON
INTSEARCHWORDCOLS R
INTSEARCHWORDCHART ON
INTSEARCHWORDFLOOR -400r
INTSEARCHWORDSORTBY REQUESTS

Again, hope this helps as well and good luck, mate!

Answer 3

For customized parsing of log files, there is a good script posted at http://www.biterscripting.com/SS_WebLogParser.html . It is written for web server logs, but you can use it as a starting point for writing your own log parser for logs of any kind.

Patrick