Kibana not recognizing Elastic Search date field

Question

I am an ElasticSearch / Kibana noob, and am trying my best to teach myself the fundamentals of this service. I am trying to insert some Keyword documents into my AWS ElasticSearch service. The document is pretty straightforward. It contains

• created_at a datetime field
• updated_at: a datetime field
• keyword: the actual keyword itself, a string such as watercolors or romanticism
• user_id: the user who created this keyword

Using the Kibana dev console, I've defined the mapping for this index using a PUT request, and then sent this GET request to check that it is valid (GET /keyword_index/_mapping/Keyword):

{
  "keyword_index": {
    "mappings": {
      "Keyword": {
        "properties": {
          "_created_at": {
            "type": "date",
            "format": "yyyy-MM-dd HH:mm:ss"
          },
          "_updated_at": {
            "type": "date",
            "format": "yyyy-MM-dd HH:mm:ss"
          },
          "keyword": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "userId": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  }
}

I've successfully inserted records into /keyword_index/Keyword, using the following POST request (from a Python script):

POST MY_ES_SERVICE_ADDRESS/keyword_index/Keyword/FzaGCiZl68/_create
{"keyword": "Comedy", "userId": "hjkJVM4TjD12N", "_created_at": "2017-12-28 02:52:14", "_updated_at": "2017-12-28 02:52:14"}

Here is a sample result from executing a match_all query:

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 332,
    "max_score": 1,
    "hits": [
      {
        "_index": "keyword_index",
        "_type": "Keyword",
        "_id": "HCQAm97kbs",
        "_score": 1,
        "_source": {
          "keyword": "Controlled",
          "userId": "",
          "_created_at": "2017-10-12 06:01:14",
          "_updated_at": "2017-10-12 06:01:14"
        }
      },  ... more results below not shown...

However, Kibana doesn't recognize my datetime fields when I attempt to create an index mapping:

Moreover, when I inspect the fields within Kibana, I see a bunch of fields I did not create mappings for, nor inserted when I sent my POST request. And, more importantly, I don't see my date fields at all:

I found a similar SO post here, but the accepted answer was to click a "refresh fields" orange button in the Kibana Management console, but I see no such thing.

Where did my date fields go? And why are they not recognized by Kibana?

Answer 1

I believe I found the culprit. Fields that begin with an underscore, such as _created_at do not show in Kibana, as I discovered in this Github issue.

I simply changed my mappings from _created_at to created_at and _updated_at to updated_at and this issue was resolved.

I don't really mind, since I understand the need to differentiate between internal and user-facing fields, but maybe some more prominent documentation within Kibana could help save us some headache and time?