I currently use Sphinx Search for optimizing our search page.
The popularity of the website is growing which causes more load on Sphinx. Although there is nothing to be worried about at this point I am wondering in how Sphinx is scalable. I have read on their website and other tutorials that it scales well but none of them really explains how exactly.
Does anyone have some concrete example on this subject?
There are 2 things :
scale on same machine - normally one core per index is used , you can split the index in several chunks ( pieces ) and create what is called a local distributed index . You also need to set dist_threads to declare how many cores to be used per search ( the best is to have more cores than chunks to leave cores free for other processes )
scale over several machines - in this case each chunk is on another server (called agent ) and one server acts as master ( or you can have several chunks per machine and use the above for each agent ) Please check http://sphinxsearch.com/docs/current.html#distributed , http://sphinxsearch.com/docs/current.html#conf-agent , http://sphinxsearch.com/docs/current.html#conf-index-type , http://sphinxsearch.com/docs/current.html#conf-dist-threads and http://sphinxsearch.com/blog/2011/10/19/dist_threads-the-new-right-way-to-use-many-cores/
