Ways to only process new(index after last run) data in Elasticsearch?

Question

Is there a way to get the date and time that an elastic search document was written?

I am running es queries via spark and would prefer NOT to look through all documents that I have already processed. Instead I would like read the only documents that were ingested between the last time the program ran and now.

What is the best most efficient way to do this?

I have looked at;

updating to add a field with an array with booleans for if its been looked at by which analytic. The negative is waiting for the update to occur.
index per time frame method, which would be to break down the current indexes into smaller ones so by hour.The negative I see is the number of open file descriptors.
??

Elasticsearch version 5.6

Add a counter to your documents and increase that each time you "process" that document. There used to be a `_timestamp` but was removed in 5.x. But still that timestamp wasn't the exact time the document has been indexed into ES ;-). — Andrei Stefan, Dec 14 '17 at 14:21

score 4 · Accepted Answer · answered Dec 14 '17 at 15:02

4

I posted the question on the elasticsearch discussion board and it appears using the ingest pipeline is the best option.

answered Dec 14 '17 at 15:02

SparkleGoat

503
1
9
22

score 2 · Answer 2 · answered Dec 18 '17 at 06:26

I am running es queries via spark and would prefer NOT to look through all documents that I have already processed. Instead I would like read the only documents that were ingested between the last time the program ran and now.

A workaround could be :

While inserting data using Logstash to Elasticsearch, Logstash appends a @timestamp key to the document which represents the time (in UTC) at which the document is created or we can use an ingest pipline

After that we can query based on the timestamp.

For more on this please have a look at :

score 1 · Answer 3 · answered Dec 11 '17 at 19:35

1

Elasticsearch doesn't have such functionality.

You need manually save with each document date. In this case you will be able to search by date range.

answered Dec 11 '17 at 19:35

Igor

149
2
9

So you are saying to add a field that holds the date/time written or now()? – SparkleGoat Dec 11 '17 at 19:36

Ways to only process new(index after last run) data in Elasticsearch?

3 Answers3