67

I was under the assumption that each shard in Elasticsearch is an index. But I read somewhere that each segment is a Lucene index.

What exactly is a segment? How does it effect search performance? I have indices that reach around 450GB in size everyday (I create a new one everyday) with default Elasticsearch settings.

When I execute curl -XPOST "http://localhost:9200/logstash-2013.03.0$i_optimize?max_num_segments=1", I get num_committed_segments=11 and num_search_segments=11.

Shouldn't the above values be 1? Maybe it's because of index.merge.policy.segments_per_tier value? What is this tier anyway?

Kamil Sindi
  • 21,782
  • 19
  • 96
  • 120
shadyabhi
  • 16,675
  • 26
  • 80
  • 131

1 Answers1

224

The word "index" gets abused a bit in Elasticsearch -- applies to too many things.

To explain:

index

An "index" in Elasticsearch is a bit like a database in a relational DB. It's where you store/index your data. But actually, that's just what your application sees. Internally, an index is a logical namespace that points to one or more shards.

Also, "to index" means to "put" your data into Elasticsearch. Your data is both stored (for retrieval) and "indexed" for search.

inverted index

An "inverted index" is the data structure that Lucene uses to make data searchable. It processes the data, pulls out unique terms or tokens, then records which documents contain those tokens. See http://en.wikipedia.org/wiki/Inverted_index for more.

shard

A "shard" is an instance of Lucene. It is a fully functional search engine in its own right. An "index" could consist of a single shard, but generally consists of several shards, to allow the index to grow and to be split over several machines.

A "primary shard" is the main home for a document. A "replica shard" is a copy of the primary shard that provides (1) failover in case the primary dies and (2) increased read throughput

segment

Each shard contains multiple "segments", where a segment is an inverted index. A search in a shard will search each segment in turn, then combine their results into the final results for that shard.

While you are indexing documents, Elasticsearch collects them in memory (and in the transaction log, for safety) then every second or so, writes a new small segment to disk, and "refreshes" the search.

This makes the data in the new segment visible to search (ie they are "searchable"), but the segment has not been fsync'ed to disk, so is still at risk of data loss.

Every so often, Elasticsearch will "flush", which means fsync'ing the segments, (they are now "committed") and clearing out the transaction log, which is no longer needed because we know that the new data has been written to disk.

The more segments there are, the longer each search takes. So Elasticsearch will merge a number of segments of a similar size ("tier") into a single bigger segment, through a background merge process. Once the new bigger segment is written, the old segments are dropped. This process is repeated on the bigger segments when there are too many of the same size.

Segments are immutable. When a document is updated, it actually just marks the old document as deleted, and indexes a new document. The merge process also expunges these old deleted documents.

slm
  • 15,396
  • 12
  • 109
  • 124
DrTech
  • 17,031
  • 5
  • 54
  • 48
  • 2
    Thanks for the detailed answer. So, how many segments are too many and or two little? In my case, a 450Gb index is created everyday. How many segments should I have? Also, can you please answer the part as to how my segments are 11+11 when I had set max_segments as 1 while optimizing. – shadyabhi Mar 15 '13 at 18:43
  • You're thinking too much about segments. Create an index with a number of shards that distributes well over the number of nodes you plan to have and let Lucene worry underneath about segments. Especially with Lucene 4 you should not be using `optimize` in the normal course of operation. – drewr Mar 16 '13 at 02:17
  • @DrewR. I am worrying because of memory usage. I've a cluster of 2 with 48GB RAM and I'm adding 450GB of data everyday so max heap is becoming a concern. I heard that optimizing indices helps in reducing memory usage. – shadyabhi Mar 17 '13 at 03:42
  • 2
    My point is not that optimization is not helpful, only to let Lucene manage it for you. With 48G nodes your ES_HEAP_SIZE should be around 24G. After that you need to figure out why you're really having heap issues. Large bulk indexing and high cardinality in your facet fields are two common issues. You may also not be taking advantage of filters properly. It's best to add monitors for your heap and correlate it with actual activity on the cluster. – drewr Mar 18 '13 at 12:48
  • So every time a new document is indexed updated in es, when a refresh/ flush happens, it would invalidate the cache created by filter?? – piyushGoyal Apr 30 '15 at 16:10
  • 4
    @piyushGoyal No - read the section in the Definitive Guide from here on http://www.elastic.co/guide/en/elasticsearch/guide/current/making-text-searchable.html#_immutability – DrTech Apr 30 '15 at 18:44
  • Aah.. This is pretty confusing. According to documentation: "Any other caches (like the filter cache) remain valid for the life of the index. They don’t need to be rebuilt every time the data changes, because the data doesn’t change. " Also it states that the Inverted Index is immutable. This means that whenever a document is updated/indexed, it is stored in ES as a new Inverted Index. If yes, then when I do a filter, obviously my request will never go the previous filter cache since it's kind of incomplete(does not have the new data), then how will it work? What's the actual flow? – piyushGoyal May 01 '15 at 05:14
  • This answer is so clear and detailed that no other answer was deemed necessary by the other SO users. Awesome! thank you :-) – asgs Jul 06 '22 at 14:17