10

  1. Could I write (with IndexWriter) new documents into index while it is opened for reading (with IndexReader)? Or must I close reading before writing?

  2. Could I read/search documents (with IndexReader) in index while it is opened for writing (with IndexWriter)? Or must I close writing before reading?

  3. Is Lucene.Net thread safely or not? Or must I write my own?

Marius Stănescu
  • 3,603
  • 2
  • 35
  • 49
Edward83
  • 6,664
  • 14
  • 74
  • 102

2 Answers2

20

You may have any amount of readers/searchers opened at any time, but only one writer. This is enforced by a directory specific lock, usually involving a file named "write.lock".

Readers open snapshots, and writers adds more data to the index. Readers need to be opened or reopened (IndexReader.Reopen) after your writer has commited (IndexWriter.Commit) the data for it to be seen, unless you're working with near-realtime-searches. This involves a special reader returned from (IndexWriter.GetReader) which will be able to see content up to the time the call to GetReader was executed. It also means that the reader may see data that will never be commited due to application logic calling IndexWriter.Rollback.

Searchers uses readers, so identical limitations on these. (Unlimited number of them, can only see what's already commited, unless based on a near-realtime reader.)

Lucene is thread-safe, and best practices is to share readers and searchers between several threads, while checking that IndexReader.IsCurrent() == true. You could have a background thread running that reopens the reader once it detects changes, create a new searcher, and then let the main threads use it. This would also allow you to prewarm any FieldCache you use to increase search speed when once the new searcher is in place.

sisve
  • 19,501
  • 3
  • 53
  • 95
  • It is useful to know about `IncRef()` and `DecRef()` and its relation to `Close()` (see docs) of `IndexReader`. They help you keep the IndexReader open as long as you still have `IndexSearcher`s referencing it (they by themselves do not call the Ref-Methods). In the scenario where you have one `IndexReader` for all searchers, this makes it easy to swap out `IndexReader`s while ensuring the open `IndexSearcher`s can finish. Please correct me, if I am wrong in any of the points. Maybe, it is not recommended anymore to have one `IndexReader` for all work. – skarmats Mar 31 '11 at 20:12
  • 1
    Those are meant for expert usage. It's often easier to never close readers directly, but let the fade away in peace and garbage collection. IndexSearcher have a hard reference to an active reader, so the reader wont be garbage collected until the searcher is. It's still best practices to reuse readers/searchers for all work (and reopen to reload changes when needed). – sisve Mar 31 '11 at 20:21
  • That is interesting. Only seeing the public .NET interface of Lucene, I always assumed, you'd have to do everything by hand. I.e. there is no `IDisposable` to be seen anywhere. The prominent `Close()` also provokes using it... I should've looked into the source code... Is it like this to keep close to the Java implementation? – skarmats Mar 31 '11 at 20:36
  • Yes. I'm not sure of the exact process, but the Lucene.NET seems to be a automatic port of Java code to C#. There's no IDisposable, because Java doesn't have it. Also, it wont follow .NET Naming Guidelines, like interfaces begin with the letter i. – sisve Mar 31 '11 at 22:20
  • You mention "creating a new" searcher if is IsCurrent is False. Would using ReOpen on the reader always be an alternative to this, or can ReOpen only be used under certain circumstances? Also, if ReOpen is used do i have to lock the reader or would this be handled internally? Put another way, can searches continue to happen on a search whose reader is currently being reopened? – Andy McCluggage Mar 15 '12 at 18:10
  • Reopen will return a new reader instance. You could leave the old instance to the garbage collector, allowing already running searches to continue with the old reader, while new searches uses the new instance. `var newReader = _reader.ReOpen(); Warmup(newReader); _reader = newReader;` – sisve Mar 16 '12 at 06:28
  • "You could have a background thread running that reopens the reader once it detects changes, create a new searcher, and then let the main threads use it." This is exactly what I need--have you come across any source code showing how this might be done? I'm a bit of a thread-newbie and I don't want to break my Lucene implementation while trying to speed it up. Thanks. – Jarvis Apr 02 '13 at 19:12
0

As I found in this mail list

Lucene.NET is thread-safe. So you can share the same instance of IndexWriter or IndexSearcher among threads. Using a write-lock, it also hinders a second IndexWriter instance to be opened with the same index.

As I see I can write and read separatly; I'll check it;)

Edward83
  • 6,664
  • 14
  • 74
  • 102