Consuming multiple data files from ES Cluster in Kibana

Question

Is it possible to consume multiple data files from an Elasticsearch cluster in real-time to create a visualization in a Kibana dashboard?

I am expecting multiple ndjson files and everytime I receive a new file I want to merge its visualization to the already existing visualization and get a combined result. The query for the expected result is :

GET output-*/_search
{
  "size": 0,
  "query": {
    "term": {
      "Field1": {
        "value": "abc"
      }
    }
  },
  "aggs": {
    "field2": {
      "terms": {
        "field": "Field2",
        "size": 10
      },
      "aggs": {
        "field3": {
          "terms": {
            "field": "Field3",
            "size": 10
          }
        }
      }
    }
  }
}

I am seeking guidance on whether it is possible to achieve the desired result with the given query and real-time data consumption.

Answer 1

You can use transform, schedule a task to run the query you shared and create a new index with the expected result.

Elasticsearch Transform