Solr requests time out during index update. Perhaps replication a possible solution?

Question

We are running a Solr installation (everything standard jetty environment, just added some fields to schema).

The index is about 80k Documents that are of average size (probably 20 fields with about 100 characters each).

The problem is that from time to time some requests time out. Well they dont time out server side but they take longer than 10 seconds which is when our application considers it as a time out. They are very simple queries that usually dont take longer than 80 ms or something.

It seems to correlate with rebuilding the index (we are gathering information from a database abnd update the index constantly in chunks of 200 documents). By constantly I mean as necessary, if no documents are there to update the indexing job is sent to sleep. I would estimate that every 15-20 minutes a commit happens.

I read the solr faqs and stuff and it seems that this is a common problem, however i did not find a solution but to increase the timeout.

But a site request that takes > 10 seconds is not acceptable.

How can i solve this? I thought about using one installatino for indexing and replicate it to another one which is used live for quering. But will this solve this problem?

Do you have any ideas on this?

score 3 · Answer 1 · answered Mar 01 '11 at 13:10

3

You're mostly on the right track. One way to solve this is to use a second core for updates, then when this second is fully updated and committed, you SWAP it with the first core and make it the active one.

I think this approach is described in more detail in the book "Solr 1.4 Enterprise Search Server" (here's a snippet)

answered Mar 01 '11 at 13:10

Mauricio Scheffer

98,863
23
192
275

thanks. in fact i have the book lying on my desk. will look into this! – The Surrican Mar 01 '11 at 13:18

score 2 · Answer 2 · answered Mar 01 '11 at 19:27

The only reason that I can think a searcher would be slow is if it is rebuilding its caches. Are you warming up your searchers with useful queries?

My thinking...

Updates in and of themselves won't block reads or writes. A commit will block writes as it flushes the writer, but not reads. One updates are flushed, a new searcher is then initialized, warmed, and then swapped out in place of the old searcher.

If at this point your searches are timing out, it could be that your first few handfuls of requests are heavily IO bound while warming up caches that later searches depend on. Hence I'm wondering if your searchers are being warmed up with useful queries.

For the sake of discussion, here's the sample newSearcher warming query present in most default-ish solrconfig.xml files:

<listener event="newSearcher" class="solr.QuerySenderListener">
  <arr name="queries">
    <lst>
      <str name="q">solr</str>
      <str name="start">0</str>
      <str name="rows">10</str>
    </lst>
    <lst>
      <str name="q">rocks</str>
      <str name="start">0</str>
      <str name="rows">10</str>
    </lst>
  </arr>
</listener>

Perhaps you're still using this? :)

Replication could be a great way to go in this kind of scenario. However, you may already have a similar mechanism in place that you can put to better use.

this looks very promising. i posted this follow up question: http://stackoverflow.com/questions/5170256/what-makes-a-good-autowarming-query-in-solr-and-how-do-they-work — The Surrican, Mar 02 '11 at 16:18

score 1 · Answer 3 · answered Mar 02 '11 at 01:39

If you only see it occasionally, and the number of documents keeps increasing, you're likely hitting the merge limit. Merging is horrendously expensive as old segments are turned into new segments, and your caches all get dumped to boot.

You definitely want to do a master/slave setup, SWAP (as seen above), etc. to smooth over the bumps.

Solr requests time out during index update. Perhaps replication a possible solution?

3 Answers3

Linked