Elasticsearch index much larger than the actual size of the logs it indexed?

Question

I noticed that elasticsearch consumed over 30GB of disk space over night. By comparison the total size of all the logs I wanted to index is only 5 GB...Well, not even that really, probably more like 2.5-3GB. Is there any reason for this and is there a way to re-configure it? I'm running the ELK stack.

Answer 1

You always have some kind of overhead when you store your logs in ES because ES will store each single line from your logs as document (in json) with added information.

Count your log lines and compare it with the count of docs in your index to find out if there is maybe a loop or a duplication of data.

Show us your elasticsearch.yml for more information and provide the info which kind of logs you are shipping to ES.

Answer 2

You can also change the mapping (or template, which generates a mapping) to remove the _source field.

http://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-source-field.html

Also, the space increase is more efficient in newver versions.

... oh, and are you looking at the total size (inclusing any replicas)