1

I want to create a date histogram with opensearch dashboards. The time format of my data is YYYY-MM-DD HH:mm:ss.SSS, which I have set under Stack Management > Advanced Settings > Date Format. I get an error like this: enter image description here

Under Discover, I can sort by "date", as it is of type "float". My field "timestamp", by which I would like to sort, is of type "string", and I cannot change this via the API:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"mapper [timestamp] cannot be changed from type [text] to [date]"}],"type":"illegal_argument_exception","reason":"mapper [timestamp] cannot be changed from type [text] to [date]"},"status":400}

I'm stuck, can someone please help?

naraghi
  • 430
  • 1
  • 6
  • 18

1 Answers1

1

To use a field for date histogram aggregation, the field type should be a date. Unfortunately, it's not possible to change the field type from Kibana => Stack management.

Here is some solution for your case:

  1. Use Histogram aggregation
  2. Set the field type and re-index the data

Here are the steps for the second option.

#Check the mapping old_index = your existing index name

GET old_index

#Put the new mapping before reindexing

PUT new_index
{
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date",
        "format": ["YYYY-MM-DD HH:mm:ss.SSS"]
      }
    }
  }
}

#reindex the data

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "old_index"
  },
  "dest": {
    "index": "new_index"
  }
}
Musab Dogan
  • 1,811
  • 1
  • 6
  • 8
  • 1
    After reindex complete check the document count and create a new index_pattern (data view) in Kibana => stack management and use that field for histogram aggregation. – Musab Dogan Jan 03 '23 at 17:33
  • 1
    Unfortunately, I am more or less bound to using opensearch. Do you know if the histogram aggregation is possible there, too? – naraghi Jan 03 '23 at 18:54
  • 1
    Yes, here is the link for histogram aggregation. https://opensearch.org/docs/latest/opensearch/bucket-agg/#histogram-date_histogram Also, the above commands will work in Opensearch too. You can go to the Opensearch Dashboard => Dev Tools and write that commands – Musab Dogan Jan 04 '23 at 07:14
  • It works the way you suggested. Thanks a lot! – naraghi Jan 04 '23 at 17:02
  • You're welcome! Happy to hear it worked. Btw, did you used histogram aggregation or did you reindex the data ? – Musab Dogan Jan 04 '23 at 17:02
  • I re-indexed the data. Although a problem I have now is that the timestamp that is supposed to be parsed from the log is being overridden by the timestamp at which the document enters opensearch. – naraghi Jan 04 '23 at 17:17
  • 1
    please open another task for it and ping me I will help you with it. you can use ingest_pipeline for that. – Musab Dogan Jan 04 '23 at 20:33
  • 1
    @Musab_Dogan I opened a new question here: https://stackoverflow.com/questions/75020349/how-to-parse-timestamp-as-date-in-opensearch – naraghi Jan 05 '23 at 14:57