We have multiple instance SOLR Cloud running, SOLR performs well but if we have spike in search request then only one node report heap issue and other node are working normal. if this happens then we need to bounce that node and heap stays normal for couple of days. if we don't bounce SOLR then eventually SOLR process will become zombie that means SOLR dashboard reports node as dead but in fact SOLR process is running in unresponsive mode and if that becomes then all search request is chocked up. we have millions of request hitting each day and we need to keep our server running 24X7 with 0 downtime.
Questions.
Main question is why only one node get high heap usage? Is there something that all queries goes to just one node? if so then why?
Thanks