Solr OutOfMemory

Question

I have Solr 4.2.1 running on Jetty 8.1.8 under Windows Server 2008 with x64 Java (JRE 7) and 2GB heap size. Solr index contains about 300-400k documents. About 3-5k new docs are added per day (mostly small, not more than 1-2kb). After about 1-2 days of work random OutOfMemoryError begin to appear on search requests, with almost same stack trace (until CompressingStoredFieldsReader.readFieldit is same as all).

I'm new to Solr and have no idea what could cause an error in that case. Sample stack trace:

at java.lang.OutOfMemoryError.<init>(<unknown string>)
at java.util.Arrays.copyOf(<unknown string>)
   Local Variable: char[]#6587
at java.lang.StringCoding.safeTrim(<unknown string>)
at java.lang.StringCoding.decode(<unknown string>)
at java.lang.String.<init>(<unknown string>)
at java.lang.String.<init>(<unknown string>)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.readField(CompressingStoredFieldsReader.java:154)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:272)
   Local Variable: org.apache.lucene.store.ByteArrayDataInput#1
   Local Variable: org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader#97
at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:139)
at org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:116)
at org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:643)
   Local Variable: org.apache.solr.search.SolrIndexSearcher$SetNonLazyFieldSelector#1
at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSolrHighlighter.java:384)
   Local Variable: java.util.HashSet#505
   Local Variable: java.lang.String[]#22
   Local Variable: org.apache.solr.common.util.SimpleOrderedMap#13
   Local Variable: org.apache.solr.search.DocSlice#15
   Local Variable: org.apache.solr.search.DocSlice$1#1
   Local Variable: org.apache.solr.highlight.DefaultSolrHighlighter#3
   Local Variable: org.apache.lucene.search.vectorhighlight.FieldQuery#1
   Local Variable: org.apache.lucene.search.vectorhighlight.FastVectorHighlighter#1
   Local Variable: org.apache.solr.search.SolrIndexSearcher#1
   Local Variable: org.apache.solr.schema.IndexSchema#2
at org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:139)
   Local Variable: java.lang.String[]#21
   Local Variable: org.apache.lucene.search.BooleanQuery#54
   Local Variable: org.apache.solr.handler.component.HighlightComponent#2
   Local Variable: org.apache.solr.common.params.DefaultSolrParams#1
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
   Local Variable: org.apache.solr.handler.component.ResponseBuilder#1
   Local Variable: java.util.ArrayList$Itr#1
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
   Local Variable: org.apache.solr.util.stats.TimerContext#1
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1817)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
   Local Variable: org.apache.solr.core.CoreContainer#1
   Local Variable: org.apache.solr.servlet.cache.Method#1
   Local Variable: org.apache.solr.servlet.SolrRequestParsers#1
   Local Variable: org.apache.solr.handler.component.SearchHandler#2
   Local Variable: org.apache.solr.response.SolrQueryResponse#1
   Local Variable: org.apache.solr.core.SolrConfig#2
   Local Variable: java.lang.String#2256
   Local Variable: java.lang.String#2255
   Local Variable: org.apache.solr.servlet.SolrRequestParsers$1#1
   Local Variable: org.apache.solr.core.SolrCore#2
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
   Local Variable: org.eclipse.jetty.servlet.ServletHandler$CachedChain#1
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
   Local Variable: org.apache.solr.servlet.SolrDispatchFilter#1
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
   Local Variable: org.eclipse.jetty.servlet.ServletHandler$CachedChain#2
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
   Local Variable: org.eclipse.jetty.server.Authentication$1#1
   Local Variable: org.eclipse.jetty.security.RoleInfo#1
   Local Variable: org.eclipse.jetty.security.ConstraintSecurityHandler#1
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
   Local Variable: org.eclipse.jetty.servlet.ServletHandler#1
   Local Variable: org.eclipse.jetty.servlet.ServletHolder#1
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
   Local Variable: org.eclipse.jetty.server.session.SessionHandler#1
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
   Local Variable: javax.servlet.DispatcherType#1
   Local Variable: org.eclipse.jetty.start.Classpath$Loader#1
   Local Variable: java.lang.String#2253
   Local Variable: java.lang.String#7243
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
   Local Variable: org.eclipse.jetty.webapp.WebAppContext#1
   Local Variable: org.eclipse.jetty.server.handler.ContextHandlerCollection#1
   Local Variable: org.eclipse.jetty.server.Handler[]#2
   Local Variable: org.eclipse.jetty.http.PathMap#1
   Local Variable: org.eclipse.jetty.http.PathMap$Entry#1
   Local Variable: org.eclipse.jetty.server.AsyncContinuation#2
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
   Local Variable: org.eclipse.jetty.server.handler.HandlerCollection#1
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:365)
   Local Variable: org.eclipse.jetty.server.Response#2
   Local Variable: org.eclipse.jetty.server.Request#2
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
   Local Variable: java.lang.String#2252
   Local Variable: org.eclipse.jetty.server.Server#1
at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
   Local Variable: org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler#2
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   Local Variable: org.eclipse.jetty.http.HttpParser#2
at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   Local Variable: org.eclipse.jetty.server.BlockingHttpConnection#2
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   Local Variable: org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint#2
at java.lang.Thread.run(<unknown string>)

You add the JVM arg `-XX:+HeapDumpOnOutOfMemoryError` and analyse the dump with a profiler. — samlewis, Jun 23 '13 at 23:49
@samlewis I did it. Stack trace is from that dumps. However I didn't find anything in that dumps that can help me. :( — lorond, Jun 24 '13 at 09:26
Can you paste some sample queries which are giving OOM errors? Most of OOM errors are caused by unoptimized queries? Also what is your index size? — AnonGeek, Jun 24 '13 at 17:11
@Saumitra It's about 6gb, 300-400k documents. My sample solr query: `{!boost b=sum(sub(1,norm(text)),recip(ms(NOW,dateCreated),3.16e-11,1,1))}(companyID:2)AND(isPublic:true OR (isPublic:false AND groupID:(242 260 395 190 176 72 67 12))) AND ((title:(test*^12)) OR (title_ft:(test^6) OR text:(test^3)))` — lorond, Jun 25 '13 at 13:07
I guess that might be because of highlighter have to reanalyze documents. I added termVectors, termPositions and termOffsets to `text` field (it's biggest one by content). Will check if it helped out in a two weeks, after deploy. — lorond, Jun 25 '13 at 13:12

Solr OutOfMemory

0 Answers0