0

We need data capacity per node based on below System & Cluster information. Could you please tell us what is (rough) data capacity per (solr) node? (without changing System & node number)

System Information per node (DSE)

CPU: 2CPUs/ 16 cores

Memory: 32 GB

HDD: 1TB

(DSE)Solr Heap Size: 15 GB

DSE Cluster information

Total Node# (all solr nodes): 4

Average Data Size per node: 24GB

Average (solr) index size per node: 11GB

Replication Factor: 2

DSE version: 4.8.3

Is Solr Heap Size (15GB) enough to support Big data (e.g 100GB solr data) during general Solr operation (e.g Query or indexing) ?

P.s: If you have any capacity formula / calculating tool please let me know.

Some programmer dude
  • 400,186
  • 35
  • 402
  • 621
Woo Kim
  • 1
  • 24GB data and 11GB index per node is not too bad. To make the searches as efficient as possible, I suggest making sure you have enough memory to cache the indexes. In your case, it looks like there will be just enough. 32GB of memory does not give you much to scale with. The thing that'll impact performance is your disk type - is that a 1TB HDD or SSD? With DSE search, you will always be index bound - that'll tell you how much memory and storage you'll need to allocate. If this data set does not grow, you should be OK. If it does, you will need more memory and possibly a larger disk. – MarcintheCloud Apr 25 '16 at 15:03
  • with current node setting, what is maximum data size or maximum solr index size per node? - (I'd like to know the timing of expanding nodes or updating resource) – Woo Kim Apr 25 '16 at 15:20
  • There isn't a hard maximum - There are practical limits though. I don't see to many Search use cases that have more than 1TB/node. That being said, look at the your index size to data size ratio - that's almost 50%. If you had 1TB of data, you would have close to 500GB of indexes. It's very hard (cost and hardware wise) to store 500GB of indexes in memory per node. 128GB memory is easier to obtain, in your case that means data per node can't exceed ~200-300GB. You can do the math from there. Again, it's all use case dependent but the goal is to keep indexes in memory to minimize search times. – MarcintheCloud Apr 25 '16 at 15:28
  • is 128GB memory in Heap or off heap (for 200~300GB data size)? How much solr heap size should be allocated among 128GB RAM to handle (200 ~300GB data size per node)? If we allocate too much of it, I expect huge GC. – Woo Kim Apr 25 '16 at 17:19
  • If Solr heap is 15Gb what is the relation between heap and index size? – Woo Kim Apr 25 '16 at 17:22

0 Answers0