How can I reduce RAM usage of Logstash?

Question

I am looking for an answer for my Logstash RAM problem because it is nearly 100%. I made a lot of searches for it but they didn't work out for me. Below code is my logstash.conf file. I think it needs small touches.

Logstash.conf:

input {
  file {
        path => ["c:/mylogs/*.txt"]
        start_position => "beginning" 
        discover_interval => 10
        stat_interval => 10
        sincedb_write_interval => 10
        close_older => 10
        codec => "json"
  }
}

filter {
  date {
        match => ["mydate","yyyy-MM-dd HH:mm:ss.SSSS" ]
        timezone => "UTC"  
  }
  date {
        match => ["TimeStamp", "ISO8601"]
  }

  json {
        source => "request"
        target => "parsedJson"
  }
}

output {
  stdout {
        codec => rubydebug
  }
  elasticsearch {
        hosts => [ "http://localhost:9200" ]
        index => "log-%{+YYYY.MM}"
  }
}

What have you tried so far? Is this particular configuration the issue? — mihomir, Feb 15 '19 at 10:05

baudsp · Answer 1 · 2019-02-18T08:45:39.043

Logstash uses a JVM to run. JVM options used by logstash can be configured in the jvm.options file, in the config folder of your logstash folder (see the doc). In the file you can set a -Xmx option to set the max heap size, which would limit the max memory used.

From the tuning logstash page, you can also configure the batch size and number of worker to reduce the number of in-flight events, which should reduce the RAM usage, but also reduce the throughput of logstash.

score 1 · Accepted Answer · answered Jul 20 '23 at 16:36

Setting Logstash JVM Options

imply setting LS_JAVA_OPTS will not work because that only appends to the default Logstash JVM Options and does not replace them. You can set JAVA_OPTS but that will not work unless you also est HEAP_DUMP_PATH. Below is the final configuration we use.

# replace logstash Java options since we only have 4 cores in production
# you will get a waring in the logs
# LS_JAVA_OPTS only appends
export JAVA_OPTS="-XX:+UseSerialGC -Djava.awt.headless=true -XX:+HeapDumpOnOutOfMemoryError"
# logstash will append Xmx
export LS_HEAP_SIZE="1g"
# this needs to be set because logstash will always append it
# if it's missing you will get an empty VM argument and it won't start
export HEAP_DUMP_PATH="-XX:HeapDumpPath=$LOGSTASH/heapdump.hprof"

How can I reduce RAM usage of Logstash?

2 Answers2