Kibana4 geo map Error - not showing the client_ip field

Question

I am trying to get kibana-4 geo map to work with ELB logs

when i click the discover tab i can clearly see a field geoip.location with values of [lat, lon] but when i click the visualise tab -> Tile map -> new search -> Geo coordinates i get an error (not showing anywhere what is the error i've also checked the kibana logs - but nothing is there)

I checked inspect element - also nothing

I then select GeoHash, but the field is empty (when i click on it its blank with a check icon)

How can i see what is the error ? How can get this map to work ?

my config is:

    input {
  file {
    path => "/logstash_data/logs/elb/**/*"
    exclude => "*.gz"
    type => "elb"
    start_position => "beginning"
    sincedb_path => "log_sincedb"
  }
}

filter {
    if [type] == "elb" {
      grok {
        match => [
          "message", '%{TIMESTAMP_ISO8601:timestamp} %{NGUSERNAME:loadbalancer} %{IP:client_ip}:%{POSINT:client_port} (%{IP:backend_ip}:%{POSINT:backend_port}|-) %{NUMBER:request_processing_time} %{NUMBER:backend_processing_time} %{NUMBER:response_processing_time} %{POSINT:elb_status_code} %{INT:backend_status_code} %{NUMBER:received_bytes} %{NUMBER:sent_bytes} \\?"%{WORD:method} https?://%{WORD:request_subdomain}.server.com:%{POSINT:request_port}%{URIPATH:request_path}(?:%{URIPARAM:query_string})? %{NOTSPACE}"'
        ]
      }

      date {
        match => [ "timestamp", "ISO8601" ]
        target => "@timestamp"
      }

      if [query_string] {
        kv {
          field_split => "&?"
          source => "query_string"
          prefix => "query_string_"
        }
        mutate {
          remove => [ "query_string" ]
        }
      }

      if [client_ip] {
        geoip {
         source => "client_ip"
         add_tag => [ "geoip" ]
        }
      }

      if [timestamp] {
        ruby { code => "event['log_timestamp'] = event['@timestamp'].strftime('%Y-%m-%d')"}
      }
    }
  }
}

output {
  elasticsearch {
    cluster => "ElasticSearch"
    host => "elasticsearch.server.com"
    port => 9300
    protocol => "node"
    manage_template => true
    template => "/etc/logstash/lib/logstash/outputs/elasticsearch/elasticsearch-template.json"
    index => "elb-%{log_timestamp}"
  }

}

Answer 1

geo_ip index did not work in my case because my index names did not started with logstash-

if you want the custom index name to get the geo-ip, you must create a template for that index name

in the output for elasticsearch use it

elasticsearch {
      manage_template => true
      template => "/etc/logstash/templates/custom_template.json"
}

your template should look like this

{
  "template" : "index_name-*",
  "settings" : {
  "index.refresh_interval" : "5s"
},
"mappings" : {
"_default_" : {
   "_all" : {"enabled" : true, "omit_norms" : true},
   "dynamic_templates" : [ {
     "message_field" : {
       "match" : "message",
       "match_mapping_type" : "string",
       "mapping" : {
         "type" : "string", "index" : "analyzed", "omit_norms" : true
       }
     }
   }, {
     "string_fields" : {
       "match" : "*",
       "match_mapping_type" : "string",
       "mapping" : {
         "type" : "string", "index" : "analyzed", "omit_norms" : true,
           "fields" : {
             "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
           }
       }
     }
   } ],
   "properties" : {
     "@version": { "type": "string", "index": "not_analyzed" },
     "geoip"  : {
       "type" : "object",
         "dynamic": true,
         "properties" : {
           "location" : { "type" : "geo_point" }
         }
      }
   }
 }
}
}

Answer 2

On our maps, we specify a field geoip.location which according to the documentation is automatically created by the geoip filter.

Can you see that field in discover? If not, can you try amending your geoip filter to

if [client_ip] {
  geoip {
     source => "client_ip"
     add_tag => [ "geoip" ]
     target => "geoip"
  }
}

and see if you can now see geoip.location in new entries?

The elasticsearch templates look for the "geoip" target when creating the associated geoip fields.

Once we have the geoip.location being created, we can create a new map with the following steps in Kibana 4.

1. Click on visualise
2. Choose 'Tile Map' from the list of visualisation types
3. Select either new search or saved - we're using a saved search that filters out Apache entries, but as long as the data contains geoip.location you should be good
4. Select the 'geo coordinates' bucket type - you'll have an error flagged at this point
5. In 'aggregation' dropdown, select 'geohash'
6. In 'field' dropdown, select 'geoip.location'