0

I have successfully snapshotted and restored data multiple times in ElasticSearch (ES) using its APIs. But now I want to merge two snapshots in ES or directly in Lucene to restore a 'larger' chunk of data.

Details: I take weekly snapshots of my data and as soon as restoration is done I delete the index so essentially the workflow looks like this

Create index abc
Snapshot index abc
Delete index abc
-----
Create index abc (again)
Snapshot index abc
Delete index abc

I have looked around but it seems there is no way to do that but those posts are an year old so wanted to reach out to the community again.

Also if not in ElasticSearch is there a way to do this Lucene directly and then configure ES to use 'new combined' index for restoration?

My language of choice for development is Python so I am looking into PyLucene as well but haven't explored it much yet.

Harshdeep
  • 5,614
  • 10
  • 37
  • 45
  • Can you explain why you delete your index `abc`? – Val Nov 14 '16 at 12:41
  • @Val I delete the index after snapshotting. So the new data is not incremental data it is entirely new batch of data/metadata. I tried looking into the files and incremental thing works only if I keep snapshotting the same index without deleting the index in between. – Harshdeep Nov 14 '16 at 12:44
  • @Val to the newer comment: I am using ES as an ad-hoc analysis store. I don't need 'historical' data in ES all the time so delete the older index. But sometimes I need 'historical' data to analyze the trend in past as well. – Harshdeep Nov 14 '16 at 12:47
  • @Harshdeep could you not use an index alias, to restore old indices,but query against all of them including your newer ones? – IanGabes Nov 14 '16 at 19:42
  • @IanGabes that is the only solution which looks decent to me at this point. I was hoping for a merging approach so that I don't have to write a very smart programmatic (abstraction) layer for restoration which renames while restoration then creates an alias. – Harshdeep Nov 15 '16 at 09:01

0 Answers0