Elasticsearch excessive memory consumption

Question

I have a Debian 10 server with 200Gb+ of disk space and 32Gb of RAM.

This server hosts two websites with a nginx + MySQL + PHP stack. Each website uses a different version of Elasticsearch (6.1.4 & 7.2.1) to index some data for autocompletion and search functionalities.

The number of documents and the disk space necessary are relatively small :

$ curl http://localhost:9721/_cat/indices?v
health status index                                                  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   xxxxxxxx_xxxxxx_prod_documents-en_us_2022-06-08-082626 eRNQbX8bS1uQGM-jSIM9-Q   1   1     157364           20     10.3mb         10.3mb
yellow close  xxxxxxxx_xxxxxx_prod_documents-en_us_2022-06-07-163138 UA-ZpoLIQZSmId8beIFPMQ   1   1                                                  
yellow open   xxxxxxxx_xxxxxx_prod_documents-fr_fr_2022-06-08-082803 eCO2D3VoRlac1MOUVJTkmA   1   1     157364         4672     20.8mb         20.8mb
yellow close  xxxxxxxx_xxxxxx_prod_documents-fr_fr_2022-06-07-163316 tlim20SmTzy8pPouPC_ngA   1   1                                                  

$ curl http://localhost:9200/_cat/indices?v
health status index                                   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   xxxxxxxx_quote_option_taxons_preprod    Ns95mRp0Ty-h3f6Y75NQoA   5   1          0            0      1.2kb          1.2kb
yellow open   xxxxxxxx_quote_full_text_prod           QBSA2OWoSx2OkBPkWnBoWw   5   1     108174           45     23.6mb         23.6mb
yellow open   xxxxxxxx_quote_full_text_preprod        l4tlrIidSe26JMg2ZGf0NA   5   1     107780            0     24.5mb         24.5mb
yellow open   xx_xxxxxxxx_shop_full_text_prod         a_g7ihFzTpWqqykaiBvriQ   5   1       1710           30      1.3mb          1.3mb
yellow open   xx_xxxxxxxx_shop_product_prod           d3vtuMGiQGKFibUAUTsARg   5   1       1710           78      1.1mb          1.1mb
yellow open   xxxxxxxx_xxxxxx_shop_product_preprod    YGIbbUtFRRyl2nHcViLgBw   5   1          0            0      1.2kb          1.2kb
yellow open   xxxxxxxx_quote_product_prod             T1u9YMtnSomBKCKP6pN6uA   5   1     108244           82     23.2mb         23.2mb
yellow open   xxxxxxxx_quote_product_preprod          whf6uxXmTJawURA9qAhHTw   5   1     107780            0     23.1mb         23.1mb
yellow open   xxxxxxxx_quote_attribute_taxons_prod    DvWW2SxnTCq530scIM4VBQ   5   1          0            0      1.2kb          1.2kb
yellow open   xx_xxxxxxxx_attribute_taxons_prod       WMLW-iKBQ9CFeVFyxt6Vrw   5   1         21            0     25.3kb         25.3kb
yellow open   xx_xxxxxxxx_option_taxons_prod          CA4nHrRlTH-JZvA8FqCkfQ   5   1          4            0     12.3kb         12.3kb
yellow open   xxxxxxxx_quote_option_taxons_prod       QrbOC8GWQOGMjfBr4xg4-Q   5   1          0            0      1.2kb          1.2kb
yellow open   xxxxxxxx_quote_attribute_taxons_preprod Y3rml_gGSSeWWRiZfgX0lw   5   1          0            0      1.2kb          1.2kb

Each elastic instance is configured with a heap size of 1Gb :

-Xms1g
-Xmx1g

However, when looking at the RAM usage, these elasticsearch instances each use aroung 9Gb of RAM :

$ ./sysmon.sh 
PID       OWNER          MEMORY         COMMAND
23883     elastic+       9149420K       /bin/java
28790     xxxxxxxx       9116236K       /var/www/elasticsearch-7.2.1/jdk/bin/java

The issue is that this server frequently runs out of RAM and starts swapping. When looking at the swap usage, elasticsearch seems to be the one taking most of it :

$ for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | head -n 1
java 317076 kB

What can explain this memory usage by elasticsearch ? Is there a way to reduce it to a size that is more coherent with the size of data that it handles ?

did you get a chance to look into my answer? – Amit Oct 06 '22 at 06:30 — Amit, Oct 06 '22 at 06:30

score 2 · Answer 1 · answered Sep 21 '22 at 10:24

2

You should check if bootstrap.memory_lock setting in your Elasticsearch is set to true, as explained in official and this blog, you should set it to false. please try changing it to false if not and see if it still takes same amount of RAM or not.

answered Sep 21 '22 at 10:24

Amit

30,756
6
57
88

1

also to add - you will likely see a lot of memory associated with Elasticsearch as the OS will be caching commonly accessed files that is uses. it's part of why Elasticsearch is so fast. you can manage this on an OS level if you really want, but you are probably not doing yourself a favour by running all of this on a single host – warkolm Sep 22 '22 at 08:13

Elasticsearch excessive memory consumption

1 Answers1