0

I have got a Splunk instance running as an HEC in Docker. I want to implement sourcetype override on a per-event basis. For that, I've added the props.conf and transforms.conf files under the $SPLUNK_HOME/etc/system/local directory.

Definitions for the files are as follows

props.conf is shown below

[source::*testing-token*]
ANNOTATE_PUNCT=false
TRANSFORMS-test_transform=test_transform_first,test_transform_second,test_transform_default

transforms.conf is shown below

[test_transform_first]
REGEX=\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}:\d{3}\]\s\[.*
DEST_KEY=MetaData:Sourcetype
FORMAT=sourcetype::mytestcustom:myservicelogs

[test_transform_second]
REGEX=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\s.*
DEST_KEY=MetaData:Sourcetype
FORMAT=sourcetype::mytestcustom:accesscombined

[test_transform_default]
REGEX=.
DEST_KEY=MetaData:Sourcetype
FORMAT=sourcetype::mytestcustom:defaultevent

When I'm sending a raw event, the HEC maps it to the sourcetype of test_transform_default even though the event is a perfect match for either test_transform_first or test_transform_second transform.

Below is a screenshot for the same enter image description here

To ensure that the regex is compliant on Splunk, I ran the below search query and got the expected result

index=testindex | regex "\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}:\d{3}\]\s\[.*"

The result is as below enter image description here

The point worth noting is that the default transform is specified as the last transform in the props.conf

What's the possible cause of the issue?

Debargha Roy
  • 2,320
  • 1
  • 15
  • 34

1 Answers1

0

The first time an event matches a sourcetype, it will fall into it

And if you tag an event with a given sourcetype in the actual HEC submission, it will always use that sourcetype

If you want something to come in differently, tag it differently in your HEC submission

warren
  • 32,620
  • 21
  • 85
  • 124
  • You mean to say the regex `.` has highest priority and thus all results (without explicit `sourcetype`) will map to it? – Debargha Roy Jan 25 '22 at 17:46
  • @DebarghaRoy - I mean `sourcetype`. If you don't explicitly say what sourcetype something is, Splunk "guesses" it's the first one in the list (in my experience) – warren Jan 26 '22 at 16:06
  • That's right. In the event of missing sourcetype, the Splunk documentation says that the way transforms are matched is in the way they are defined. So technically, I would expect the default to be executed last as it's in the last of the list `TRANSFORMS-test_transform=test_transform_first,test_transform_second,test_transform_default`. But that's what is not happening with events which match the `*first`/`*second` transform. – Debargha Roy Jan 26 '22 at 17:50
  • @DebarghaRoy - looks like you're getting a match in lexicographic order ("default" being lexicographically prior to "first" or "second"). Try renaming it to be lexicographically-*last* ... eg "`zzz-test_transform_default`" – warren Jan 26 '22 at 18:24
  • Looks like the issue was with the Splunk instance. Performing a clean setup worked. – Debargha Roy Feb 09 '22 at 13:49