0

Is it possible, using elasticsearch and mongodb river, to import documents which are created after a specified date (documents includes a timestamp) ?

Stennie
  • 63,885
  • 14
  • 149
  • 175
smace
  • 1,088
  • 2
  • 11
  • 16
  • How about a [range query](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-range-query.html) on the `mongoindex`? – Srikanth Venugopalan Sep 17 '14 at 04:45
  • This isn't what I'm looking for, I have 10 millions documents per day in mongo, and a small ES which can't handle the whole data! – smace Sep 18 '14 at 08:23

1 Answers1

0

It's been a while since the question was posted, but just in case someone else finds it useful. You can use a custom filter when creating the river, in the mongodb section of the request,

mongodb : {
    ...
    "filter": ${mongo.filter},
    ...
}

this way, the river will only index those documents that match the filter condition, however there is a limitation, according to the docs, the river won't remove the documents that at some point in the future stop matching the filter. See more info here.

A somewhat more complicated way to accomplish this is by using the initial_timestamp option, in which you provide a script (javascript) that defines the timestamp for the initial document import, see an example here. It's all in the project wiki.

Camilo
  • 1,889
  • 2
  • 17
  • 16