0

I tried to find information about this, but I have not find what I was looking for.

I am storing metrics every minutes in an Elasticsearch database. My idea is that the frequency is important only in a short period.

For example, I want to have my metrics every minutes for the last past week, but then I would like to merge these metrics in order to have only one document of metrics for each past weeks.

Thus, I have an idea to achieve this with a stream processing framework such as Spark streaming or Flink, but my question is : is there a native way / tool / tricks to make it happen in Elasticsearch ?

Thank you, hope my question is clear enough, otherwise leave a comment for more details.

ImbaBalboa
  • 851
  • 9
  • 23
  • 1
    One idea would be to have a weekly index in which you store all your metrics every minutes, once the week has passed, you could run an aggregation query on the past week index and aggregate all info at the day or week level. You'd then store that weekly aggregated information as new document in another historical index that you can query later on. I don't think it's necessary to leverage Spark streaming for this, ES aggregations can do the job pretty easily. – Val Apr 14 '17 at 15:22
  • Thanks for the comment, I think this is what I was looking for. You can make it an answer. – ImbaBalboa Apr 14 '17 at 15:50

1 Answers1

0

One idea would be to have a weekly index in which you store all your metrics every minutes, once the week has passed, you could run an aggregation query on the past week index and aggregate all info at the day or week level. You'd then store that weekly aggregated information as new document in another historical index that you can query later on. I don't think it's necessary to leverage Spark streaming for this, ES aggregations can do the job pretty easily.

Val
  • 207,596
  • 13
  • 358
  • 360