IIS Logs and Event Logs

Question

First off thank you for any advice and your time.

I recently setup an Elk stack for the company I just started working for. (This is my first experience using Logstash and Nxlog.) What I would like to do is send both IIS logs and EventLogs from the same webserver to logstash using nxlog.

I just don't understand how to send two types of logs from one source and have the logstash.conf filter this data correctly.

This is my nxlog.conf

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension json>
    Module xm_json
</Extension>
<Input iis_1>  
      Module    im_file
      File    "F:\inetpub\logs\LogFiles\W3SVC1\u_ex*.log"
    ReadFromLast True
    SavePos True
    Exec    if $raw_event =~ /^#/ drop();
</Input>  
<Input iis_2>  
      Module    im_file
      File    "F:\inetpub\logs\LogFiles\W3SVC2\u_ex*.log"
    ReadFromLast True
    SavePos True
    Exec    if $raw_event =~ /^#/ drop();
</Input>
<Input iis_4>  
      Module    im_file
      File    "F:\inetpub\logs\LogFiles\W3SVC4\u_ex*.log"
    ReadFromLast True
    SavePos True
    Exec    if $raw_event =~ /^#/ drop();
</Input>  
<Input eventlog>
        Module im_msvistalog
        Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>
<Output out_iis>  
    Module  om_tcp
    Host    10.191.132.86
    Port    5555
    OutputType  LineBased
</Output>  
<Route 1>  
    Path    iis_1, iis_2, iis_4, eventlog=> out_iis
</Route>

My Current logstash.conf

input {  
      tcp {
              type => "iis"
              port => 5555
              host => "10.191.132.86"
      }
}
filter {  
    if [type] == "iis" {
        grok {
            match => ["@message", "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:query} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:referrer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}"]
        }
    }
}
output {  
    elasticsearch {
    protocol => "http"
        host => "10.191.132.86"
        port => "9200"
    }
}

It looks like you can filter different data by setting the type and doing if type else this type. But if they are coming from the same source how do I specify different types?

:) Thanks!

score 1 · Answer 1 · answered Apr 18 '15 at 10:14

1

NXLog sets the field SourceModuleName with the value iis_1, iis_2, etc. You may want to use this instead.

answered Apr 18 '15 at 10:14

b0ti

2,319
1
18
18

Thanks for this tip, I searched around for an example of syntax using this and I could not find anything. (That was clear at least.) Do you by chance have a small example? – Paul Shwag Apr 20 '15 at 15:44

score 0 · Accepted Answer · answered Apr 17 '15 at 19:27

0

A way to do this is filter by a known record entry in each log and wont exist in the other, for example [cs_bytes etc]:

e.g.

if [iisfield] {
   mark type as IIS
else 
   mark type as EventLog
}

I have written a IIS and Event log agent that captures logs for Logit.io they might already do everything you already want

answered Apr 17 '15 at 19:27

Lee Smith

69
4

Awesome, I will try that out. Thank you. Do you know if I can have multi outputs like this? I have been messing around with it this afternoon when I have time. (Yet to be successful haha.) Module om_tcp Host 10.191.132.86 Port 5555 OutputType LineBased Module om_tcp host 10.191.132.86 Port 5556 Path iis_1, iis_2, iis_4=> out_iis Path eventlog=> out_iis2 – Paul Shwag Apr 17 '15 at 20:50
Yes you can have multiple outputs in your config, it will copy each log to every output, you can also do the same with inputs, however all the filters run against each log you pass through – Lee Smith Apr 17 '15 at 21:10
Also be aware, that logstash will pick any file up in the config directory, so if you have a copy, it will run them twice etc, you can split you configs down by file e.g. 00_inputs.conf, 01_filters.conf, 02_outputs.conf, the number is important as it reads them in order – Lee Smith Apr 17 '15 at 21:11
If you add the type into the output 'output { if [type] == "iis" { elasticsearch { protocol => "http" host => "10.191.132.86" port => "9200" } } }' This will ensure you get only the type of logs you want sent to the correct output – Lee Smith Apr 17 '15 at 21:14
Awesome, I will try these out today! – Paul Shwag Apr 20 '15 at 15:37

IIS Logs and Event Logs

2 Answers2