2
{"level":"error","app":"my-app","component":"my-component","message":"my-message"}

I have plenty of log messages like this one above.

In order to select such events in Splunk Search, I use the query below (the names have been changed but the idea is the same):

index="my-index" app="my-app" message="my-message" component="my-component"
|  stats count

For a reason I don't understand, this query returns 0 rows which doesn't make any sense unless I misunderstand the syntax. Strangely enough, component is the only field that behaves this way. There are many more fields in my log messages and adding as many of them as I want using the same ... field1=value field2=value syntax works fine.

The query works, however, with either |where component IN ("my-component") or |spath component | search component="my-component".

I would think that my syntax is wrong but it only returns strange results with one of the fields. What could be wrong in this case?

Update: It's also interesting how the query below returns a valid breakdown of events by component:

index="my-index" app="my-app" | stats count by component

The query below, however, still returns 0.

index="my-index" app="my-app" component="my-component" | stats count by component
Don Draper
  • 463
  • 7
  • 21
  • Does the component field contain any major breaker characters? If so, Splunk is not searching for what you think it's searching for. Open the Job Inspector and look at the litsearch to see how it interpreted your query. – RichG Jan 31 '23 at 15:04
  • This is what gets generated in the inspector: `search index="my-index" app="my-app" component="my-component"` Nothing suspicious to be honest – Don Draper Jan 31 '23 at 15:18
  • What do you get for `index="my-index" component="my-component"`? Just want to exclude typos somewhere else. And also, `index="my-index" component=* | stats count by component`. – PM 77-1 Jan 31 '23 at 15:18
  • Quite surprisingly, I get 0 rows too with such a request. As long as there is this "component" field, it doesn't matter what the rest of the fields are. As I wrote above, using ` | where` or `spath` solves the issue (with the same spelling of course) but I don't understand why – Don Draper Jan 31 '23 at 15:23
  • In the Job Inspector, scroll down to "Search job properties" and look for "optimizedSearch" and "normalizedSearch" to see how Splunk is running the query. Compare those fields with those from searches that use `| where...`. – RichG Jan 31 '23 at 15:38
  • `optimizedSearch: | search (app="my-app" component="my-component" index="my-index")` The `litsearch` field is a mess for some reason: it has plenty of `!=` filters but the value of my `component `filter is correct – Don Draper Jan 31 '23 at 15:43
  • Querying `... component=* | stats count by component` returns 0 rows but `... | stats count by component` shows correct data – Don Draper Jan 31 '23 at 15:46
  • 1
    You may want to add your `stats .. by` results to your question. – PM 77-1 Jan 31 '23 at 16:03
  • I agree. Updated the question – Don Draper Jan 31 '23 at 16:21
  • what does your data *actually* look like? It sounds like you've got some unescaped characters in your JSON – warren Feb 01 '23 at 13:20
  • Is there a fields.conf entry for `component`, or a props/transforms entry that applies to the index/source/sourcetype/etc that may be confusing Splunk here? Is that field the only one not indexed when the other fields are indexed (or vice versa)? – dskrypa Feb 02 '23 at 01:20

0 Answers0