Splunk: How Do I Extract Fields from W3C Extended Format

Question

I'm trying to configure Splunk to properly parse the fields of the W3C log format.

Now, I'm running into configuration confusion: where and how do I specify how to split up the log format?

My Inputs.conf looks like this:

[monitor://C:\WINDOWS\system32\LogFiles\W3SVC98989898]
disabled = false
host = mywebsite.net
sourcetype = iis

I tried adding this to my sourcetypes.conf:

[iis_w3c_default]
DELIMS = " "
FIELDS = "date", "time", "cs-method", "cs-uri-stem", "cs-uri-query", "cs-username", "c-ip", "cs-version", "cs(User-Agent)", "cs(Referer)", "sc-status", "sc-bytes"

But using ths as a sourcetype failed to extract any meaningful fields.

I guess I just don't grasp everything that Splunk is doing...

Once i have this working, I plan on reindexing all of my data with the new fields (which will be an ordeal in-and-of-itself).

I really, really want to like Splunk but configuration is just too wonky...

Answer 1

You can filter the lines beginning with a # using a transform.

In props.conf (you can just add the extra line below your existing setup):

[iis_w3c_default]
TRANSFORMS-blacklist-hash = iis_blacklist_hash

In transforms.conf:

[iis_blacklist_hash]
REGEX = ^#
DEST_KEY = queue
FORMAT = nullQueue

Answer 2

Here is what I did:

etc\system\local\props.conf:

[iis_w3c_default]
REPORT-foobar=iis_w3c_default_extractions

etc\system\local\transforms.conf:

[iis_w3c_default_extractions]
DELIMS = " "
FIELDS = "date", "time", "cs-method", "cs-uri-stem", "cs-uri-query", "cs-username", "c-ip", "cs-version", "cs(User-Agent)", "cs(Referer)", "sc-status", "sc-bytes"

Will update once I figure out how to remove lines that have a "#" symbol in them.