Intermittent connection timeouts to Solr server using SolrNet

Question

I have a production webserver hosting a search, and another machine which hosts the Solr search server (on a subnet which is in the same room, so no network problems). All is fine >90% of the time, but I consistently get a small number of The operation has timed out errors.

I've increased the timeout in the SolrNet init to 30 seconds (!)

SolrNet.Startup.Init<SolrDataObject>(
    new SolrNet.Impl.SolrConnection(
        System.Configuration.ConfigurationManager.AppSettings["URL"]
    ) {Timeout = 30000}
);

...but all that happened is I started getting this message instead of Unable to connect to the remote server which I was seeing before. It seems to have made no difference to the amount of timeout errors.

I can see nothing in any log (believe me I've looked!) and clearly my configuration is correct because it works most of the time. Anyone any ideas how I can find more information on this problem?

EDIT:

I have now increased the number of HttpRequest connections from 2 to 'a large number' (I see up to 10 connections) - but this has had no discernible effect on this problem.

The firewall is set to allow ANY connections between the two machines.

We've also checked the hardware with our server host and there are no problems on the connections, according to them.

EDIT 2:

We're still seeing this issue.

We're now logging the timeouts and they're mostly just over 30s - which is the SolrNet layer's timeout; some are 20s, though - which is the Tomcat default timeout period - which suggests it's something in the actual connection between the machines.

Not sure where to go from here, though - they're on a VLAN and we're specifically using the VLAN address - response time from pings is ALWAYS <1ms.

Hope this answers your question: We do a dataimporthandler delta once every 15 minutes; and a full import overnight. But I don't do any kind of optimization. I'm going to go and read up now... What would you suggest? — Ian Grainger, Jan 09 '12 at 09:05
Hmm, had a read, and have just added a scheduled update?optimize=true every night after the full import... But TBH I don't believe that optimization's going to do much considering the timeout is over 30s and usually this returns in much less than a second. — Ian Grainger, Jan 09 '12 at 09:24

score 2 · Answer 1 · edited May 23 '17 at 11:51

2

Without more information, I can only guess a few possible reasons:

You're fetching tons of documents per query, and it times out while transferring data.
You're hitting the ServicePoint.ConnectionLimit. If so, just increase this value. See also How can I programmatically remove the 2 connection limit in WebClient
You have some very facet-heavy requests or misusing Solr (e.g. not using filter queries). Check the qtime in the response. See the Solr performance wiki for more details.

edited May 23 '17 at 11:51

Community

1
1

answered Jan 09 '12 at 13:04

Mauricio Scheffer

98,863
23
192
275

I didn't copy an example querystring because it's huge, lots of FQs and lots of Facet.Queries. QTime is down in the 100-200s, though. I've got rows set to 20, so not returning tons of documents. Thanks for the links. Will check them. – Ian Grainger Jan 09 '12 at 14:44
I'm interested in changing the ServicePoint.ConnectionLimit - but not sure the best way to do this using SolrNet? Just put it up top: System.Net.ServicePointManager.DefaultConnectionLimit = 200; SolrNet.Startup.Init(new SolrNet.Impl.SolrConnection(System.Configuration.ConfigurationManager.AppSettings["URL"]) { Timeout = 30000 }); ? – Ian Grainger Jan 09 '12 at 14:59
@IanGrainger yes, alternatively you can set it in your web.config. – Mauricio Scheffer Jan 09 '12 at 15:00
OK, I tried setting the limit to 200 using the web.config , but looking on the web server machine with CurrPorts.exe it still seems to only have 2 connections to my search server (we're using IIS in classic mode) - so this could well be the issue! I don't need to restart IIS or something, do I? – Ian Grainger Jan 10 '12 at 10:05
Weird. Our other servers can create more than 2 connections regardless of this setting, but not this one... – Ian Grainger Jan 10 '12 at 11:53

Dorin · Answer 2 · 2013-11-08T14:00:36.210

0

Try setting this in .net.

ServicePointManager.Expect100Continue = false;

or this

ServicePointManager.SetTcpKeepAlive(true, 200000, 200000); - this sends requests to the server to keep the connection alive.

edited Nov 08 '13 at 14:00

answered Nov 08 '13 at 11:05

Dorin

2,482
2
22
38

After quite extensive reading around - I think this might be the way that Java is running. I think it stops servicing requests while it recycles memory or something similar ... I will investigate these the next time I make a change to the server, though - thanks for the suggestions. – Ian Grainger Nov 08 '13 at 16:41

Intermittent connection timeouts to Solr server using SolrNet

2 Answers2

Linked