6

I am googling for two days... I really need help.

I have an application with multiple threads trying to update a lucene index using Open/Close of IndexWriter for each update. Threads can start at any time.

Yeah, the problem with write.lock! So there are may be two or more solutions:

1) check IndexWriter.IsLocked(index) and if it is locked to sleep the thread.
2) Open an IndexWriter and never close it. The problem is that I have another application using the same index. Also when should I close the index and finalize the whole process?

here are interesting postings
Lucene IndexWriter thread safety
Lucene - open a closed IndexWriter

Update: Exactly 2 years later I did a rest API which wraps this and all writes and reads were routed to the API.

Community
  • 1
  • 1
mynkow
  • 4,408
  • 4
  • 38
  • 65

1 Answers1

7

Multiple threads, same process:

Keep your IndexWriter opened, and share it across multiple thread. The IndexWriter is threadsafe.

For multiple processes, your solution #1 is prone to issues with race conditions. You would need to use named Mutex to implement it safely: http://msdn.microsoft.com/en-us/library/bwe34f1k.aspx

That being said, I'd personally opt for a process dedicated to writing to the index, and communicate with it using something like WCF.

Jf Beaulac
  • 5,206
  • 1
  • 25
  • 46
  • Yep, I was thinking about something like external process handling the index updates. What are the consequences of not closing the index writer but doing only commit? Is there something I should consider to do manually regarding IndexReader and index fragment merge? – mynkow Sep 11 '12 at 18:33
  • It all depends on the volume of new documents and the MergePolicy you use. Usually you'll want to Optimize (IndexWriter.Optimize()) on low traffic hours. Keeping the IndexWriter opened is usually better if your Searchers are created in the same process, because you can use near real time search: http://wiki.apache.org/lucene-java/NearRealtimeSearch – Jf Beaulac Sep 11 '12 at 18:45
  • What about the searchers in different process? – mynkow Sep 11 '12 at 19:12
  • You will need to detect changes to the index and re-open them, or periodically re-open them to see changes. – Jf Beaulac Sep 11 '12 at 20:58
  • 1
    Exactly 2 years later I did a rest API which wraps this: https://github.com/NielsKuhnel/NrtManager and all writes and reads were routed to the API. – mynkow Sep 11 '14 at 14:37