How to migrate data between two elasticsearch clusters

Question

I've been searching the web for the last couple of hours trying to figure this out. I have a central ES server that consumes all my data. It sits in a private network. On a separate network I want to bring up another ES and give a clients access to it, so they can check some stats.

My questions:

What's the best way to feed the data to this new instance based on a query result that is specific to this client?
In the case of Logstash, why doesn't it go through all the indexes returning all documents that match my query?

Eg.:

"query": { 
    "bool": { "must": { "match": { "referrer": "clientsite.com" }}} 
}

From my research, the new "Reindex from Remote" feature in ES 5.0 is the way of the future, but most internet articles point to Logstash using elasticsearch plugin as input and output method.

I tried that but it seems like logstash only produces a small amount of results and then quits. if I use only "match_all": {}" it seems to go through all my documents.

Here is my logstash.conf file:

input {
  elasticsearch {
    hosts => "192.168.0.1"
    index => 'logstash-*'
    query => '{ "query": { "bool": { "must": { "match": { "referrer": "clientsite.com" } } } } }'
    scroll => "1m"
    size => 1000
  }
}

  stdout {
    codec => "rubydebug"
  }
}

I also tried this query line, with no success:

query => '{ "query": { "bool": { "must": { "match_all": {} } , "filter": { "term": { "referrer": "uol.com.br" } } } },  "sort": [ "_doc" ] }'

It sounds that you need to some filter in your query (by creation date or by ID if it's int). — Adam Łepkowski, Nov 05 '16 at 21:54
In my final query I did add a date filter to control the slice of time I wanted replicated. Thanks for the tip though. — Luiz Oliveira, Nov 08 '16 at 19:21

score 0 · Answer 1 · answered Nov 07 '16 at 16:28

0

Logstash was behaving as expected. What made logstash do what I wanted it to was making a change from "match" to "wildcard".

With this line:

query => '{ "query": { "bool": { "must": { "wildcard": { "referrer": "*clientsite.com*" } } } } }'

Now I'm able to retrieve all the documents.

Hope it helps someone else.

answered Nov 07 '16 at 16:28

Luiz Oliveira

1

using a wildcard is not a good practice because this type of query is executed on a not_analyzed field and it can make a performance issue when you work with a big data set. – Adam Łepkowski Nov 08 '16 at 20:59

How to migrate data between two elasticsearch clusters

1 Answers1