How to search on a URL exactly in ElasticSearch / Kibana

Question

I have imported an IIS log file and the data has moved through Logstash (1.4.2), into ElasticSearch (1.3.1) and then being displayed in Kibana.

My filter section is as follows:

filter {
  grok {
     match => 
        ["message" , "%{TIMESTAMP_ISO8601:iisTimestamp} %{IP:serverIP} %{WORD:method} %{URIPATH:uri} - %{NUMBER:port} - %{IP:clientIP} - %{NUMBER:status} %{NUMBER:subStatus} %{NUMBER:win32Status} %{NUMBER:timeTaken}"]
  }
}

When using a Terms panel in Kibana, and using "uri" (one of my captured fields from Logstash), it is matching the tokens within the URI. Therefore it is matching items like:

'Scripts'
'/'
'EN

Q: How do I display the 'Top URLs' in their full form?

Q: How do I inform ElasticSearch that the field is 'not_analysed'. I don't mind having 2 fields, for example:

uri - The tokenized URI
uri.raw - the fully formed URL.

Can this be done Logstash side, or is this a mapping that needs to be set up in ElasticSearch?

Mapping is as follows :

//http://localhost:9200/iislog-2014.10.09/_mapping?pretty

{
  "iislog-2014.10.09" : {
    "mappings" : {
      "iislogs" : {
        "properties" : {
          "@timestamp" : {
            "type" : "date",
            "format" : "dateOptionalTime"
          },
          "@version" : {
            "type" : "string"
          },
          "clientIP" : {
            "type" : "string"
          },
          "device" : {
            "type" : "string"
          },
          "host" : {
            "type" : "string"
          },
          "id" : {
            "type" : "string"
          },
          "iisTimestamp" : {
            "type" : "string"
          },
          "logFilePath" : {
            "type" : "string"
          },
          "message" : {
            "type" : "string"
          },
          "method" : {
            "type" : "string"
          },
          "name" : {
            "type" : "string"
          },
          "os" : {
            "type" : "string"
          },
          "os_name" : {
            "type" : "string"
          },
          "port" : {
            "type" : "string"
          },
          "serverIP" : {
            "type" : "string"
          },
          "status" : {
            "type" : "string"
          },
          "subStatus" : {
            "type" : "string"
          },
          "tags" : {
            "type" : "string"
          },
          "timeTaken" : {
            "type" : "string"
          },
          "type" : {
            "type" : "string"
          },
          "uri" : {
            "type" : "string"
          },
          "win32Status" : {
            "type" : "string"
          }
        }
      }
    }
  }
}

Its the default out of the box. No additional mappings have been creatied — Dominic Zukiewicz, Oct 14 '14 at 09:56

score 1 · Answer 1 · answered Oct 14 '14 at 09:55

1

In your Elasticsearch mapping:

url: {
  type: "string",
  index: "not_analyzed"
}

answered Oct 14 '14 at 09:55

Pierre-Louis Gottfrois

17,561
8
47
71

Will I need to add this to every index? The log files span 2 months? – Dominic Zukiewicz Oct 14 '14 at 09:58
You might want to read this http://www.elasticsearch.org/blog/new-in-logstash-1-3-elasticsearch-index-template-management/ – Pierre-Louis Gottfrois Oct 14 '14 at 10:07
More here https://github.com/elasticsearch/logstash/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template.json – Pierre-Louis Gottfrois Oct 14 '14 at 10:09

score 0 · Answer 2 · answered Oct 14 '14 at 12:48

The problem is that the iislog- is not compliant with the logstash- format, and hence did not pick up the template:

My index format was iislog-YYYY.MM.dd, this did not use the out-of-the-box mappings by Logstash. When using the logstash- index format, Logstash will create 2 pairs of fields for strings. For example uri is:

uri (appears in Kibana)
uri.raw (does not appear in Kibana)

Note that the uri.raw will not appear in Kibana - but it is queryable.

So the solution to use an alternative index is to:

Don't bother! Use the default index format of logstash-%{+YYYY.MM.dd}
Add a "type" to the file input to help you filter the correct logs in Kibana (whilst using the logstash- index format)
```
input { 
  file {
      type => "iislog"
      ....
  }
}
```
Apply filtering in Kibana based in the type

OR

If you really really do want a different index format:

Copy the default configuration file to a new file, say iislog-template.json

Reference the configuration file in the output ==> elasticsearch like this:

output {    
   elasticsearch_http {
      host => localhost
      template_name => "iislog-template.json"
      template => "<path to template>"
      index => "iislog-%{+YYYY.MM.dd}"   
   }
}

How to search on a URL exactly in ElasticSearch / Kibana

2 Answers2