Huge database indexing elasticsearch

Question

I'm trying to index a database with over 1 million posts using elasticsearch the problem I'm having is that due to the amount of data the indexing crashes

is it possible to do the indexing in parts or is this something that should not crash and I'm just doing something wrong?

I'm using Elastica (https://github.com/ruflin/Elastica)

---- edit ----

forgot to add some errors, this is what I get and after that nothing happens

Caused by: org.apache.lucene.queryparser.classic.ParseException: Encountered "<EOF>" at line 1, column 5.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <NUMBER> ...

    at org.apache.lucene.queryparser.classic.QueryParser.generateParseException(QueryParser.java:708)
    at org.apache.lucene.queryparser.classic.QueryParser.jj_consume_token(QueryParser.java:590)
    at org.apache.lucene.queryparser.classic.QueryParser.Clause(QueryParser.java:275)
    at org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:181)
    at org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:170)
    at org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:121)
    ... 18 more

Please add details about the "crash". Like this it could be anything from client running out of time to running out of diskspace or computer memory. — snitch182, Aug 14 '13 at 12:27

score 0 · Accepted Answer · answered Aug 14 '13 at 12:48

0

Ok I found my error, the problem was that I was loading to much data to fast

just adding a sleep(1); solved the problem

answered Aug 14 '13 at 12:48

Touchpad

662
2
12
29

1

The error sound like an socket timeout not catched, resulting in an incomplete http call, anyway, hope you use Bulk indexing! – Damien Aug 20 '13 at 07:43
Thanks for this response, I've rebuilt my code like 3 times since this and I now have a working way of doing it, yes I use bulk but thanks for the tip =) – Touchpad Aug 23 '13 at 11:50

Huge database indexing elasticsearch

1 Answers1