0

I am using solr 4.6.0 with jetty on windows 7 enterrpise with max heap of 2G.I can do a full-import for 200,000 records properly from the Solr Admin UI but as soon as I increase to 250,000 records, it starts giving me this error below:

webapp=/solr path=/dataimport params={optimize=false&clean=false&indent=true&commit=true&verbose=true&entity=files&command=full-import&debug=true&wt=json&rows=250000} {add=[8065121, 8065126, 8065128, 8065146, 8065963, 7838189, 7838186, 8065155, 8065174, 8065179, ... (250001 adds)],commit=} 0 2693420

org.apache.solr.common.SolrException; null:org.eclipse.jetty.io.EofException
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)
at org.eclipse.jetty.http.AbstractGenerator.blockForOutput(AbstractGenerator.java:507)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:170)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
at su

Caused by: java.net.SocketException: Software caused connection abort: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method)
at j......  
org.apache.solr.common.SolrException;null:org.eclipse.jetty.io.EofException at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)
org.eclipse.jetty.servlet.ServletHandler; /solr/dihdb/dataimport
java.lang.IllegalStateException: Committed
at org.eclipse.jetty.server.Response.resetBuffer(Response.java:1144)

I have changed example/etc/jetty.xml as follows for maxIdleTime=3500000.

I changed example/etc/webdefault.xml for session-timeout=720.

I still keep getting the error above.

TIA,

Vijay

Vijay
  • 595
  • 1
  • 13
  • 27
  • Please give some more details. Are you indexing data to Solr from a db through DataImportHandler? – spyk Jan 17 '14 at 14:54
  • I have pulled the data from a DB into a csv file locally with 250K-500K records.I am trying to import data from this local csv file into Solr. – Vijay Jan 17 '14 at 17:24
  • I am using FileListEntityProcessor and LineEntityProcessor to import the csv data. – Vijay Jan 17 '14 at 17:30
  • Saw a similar issue which recommends sending lower no of docs to be committed. http://web.archiveorange.com/archive/v/AAfXfyoBz7lVcVeQgx6z – Vijay Jan 17 '14 at 20:25
  • This could also help: http://stackoverflow.com/questions/9449185/solr-best-approach-to-import-20-million-documents-from-csv-file – spyk Jan 17 '14 at 20:43
  • Also tried 100000 1800000 but that didn't work either. – Vijay Jan 17 '14 at 22:19
  • I changed -Xmx5120M and that seems to have fixed the issue with 500K records.Lack of memory in essence was the issue for this misleading error showing up. – Vijay Jan 19 '14 at 03:26

1 Answers1

0

I changed -Xmx5120M and that seems to have fixed the issue with 500K and 1 million records.Lack of memory in essence was the issue for this misleading error showing up.

Also tried 100000 1800000 for DataImportHandler.

Vijay
  • 595
  • 1
  • 13
  • 27