0

I have a single node ElasticSearch cluster that has one index (I know, my bad) where I inserted 2B documents.

I did not know it was a best practice to split indices and mine grew to 400GB before it crashed.

I tried splitting my index with (https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-split-index.html) and I keep getting java.lang.OutOfMemoryError no matter what I do. I have maxed out my physical memory and threads just got stuck in the _split.

I had some files that were deleted via logstash when they were successfully indexed, so reinserting the data is not an option.

Any suggestions?

patito
  • 530
  • 2
  • 13
  • May I ask you what do you want to achieve with `_split`? Are other operations on the index successful (search, insert)? – Nikolay Vasiliev Jul 04 '19 at 15:05
  • I'm trying to separate the index into a couple of shards so I can query it, at the moment I can't perform any operation on it. The split decision comes from me trying not to load 400GB of memory to query. Is this not the right procedure? – patito Jul 04 '19 at 16:18
  • This situation looks very serious. How many shards do you have at the moment? Which version of Elasticsearch do your run on? – Nikolay Vasiliev Jul 04 '19 at 18:48
  • I had a single shard setup due to logstash creating the index. I'm running the latest stable release 7.0.2 – patito Jul 05 '19 at 16:41
  • Can you spawn another Elasticsearch instance? How much memory does it already use, how much memory is there available on the machine? – Nikolay Vasiliev Jul 05 '19 at 20:56

2 Answers2

0

Add swap space or increase RAM of that server.

I'm still confused as to where you got 2 Billion documents :/

Nithin
  • 1,387
  • 4
  • 19
  • 48
0

Never use swap in ES Machines ,

Use https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-recovery.html to check status of split

also do you changed the max memory option in jvm config for ES - https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html

midhuntp
  • 1
  • 2
  • I changed the configurations and monitored the split, I still can't complete the split job and my index is unusable. – patito Jul 03 '19 at 22:54