Spring Boot Integration Test failes due to lucene lock when using Hibernate Search

Question

In my Spring Boot 1.5.10.Final project I use Hibernate Search ORM 5.6.4.Final. It works fine except of the integration tests. There is one test class with several test methods to test the search logic. If I run just this test class every things works fine. Spring Boot is starting and creates the index. If I run this test class together with all other integration tests, every test class will throw an LockObtainFailedException and the Hibernate Search tests will fail.

org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: ...LieferantEntity\write.lock
    at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:127) ~[lucene-core-5.5.5.jar:5.5.5 b3441673c21c83762035dc21d3827ad16aa17b68 - sarowe - 2017-10-20 08:57:09]
    at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-5.5.5.jar:5.5.5 b3441673c21c83762035dc21d3827ad16aa17b68 - sarowe - 2017-10-20 08:57:09]
    at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-5.5.5.jar:5.5.5 b3441673c21c83762035dc21d3827ad16aa17b68 - sarowe - 2017-10-20 08:57:09]
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:776) ~[lucene-core-5.5.5.jar:5.5.5 b3441673c21c83762035dc21d3827ad16aa17b68 - sarowe - 2017-10-20 08:57:09]
    at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.createNewIndexWriter(IndexWriterHolder.java:126) ~[hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
    at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.getIndexWriter(IndexWriterHolder.java:92) ~[hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
    at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriter(AbstractWorkspaceImpl.java:117) ~[hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
    at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.getIndexWriterDelegate(AbstractWorkspaceImpl.java:203) ~[hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
    at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:81) [hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
    at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:46) [hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
    at org.hibernate.search.backend.impl.lucene.SyncWorkProcessor$Consumer.applyChangesets(SyncWorkProcessor.java:165) [hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
    at org.hibernate.search.backend.impl.lucene.SyncWorkProcessor$Consumer.run(SyncWorkProcessor.java:151) [hibernate-search-engine-5.6.4.Final.jar:5.6.4.Final]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]

I use the default settings. If I set exclusive_index_use to false, then it works without failure but then the test execution is very slow. For me it seems the index is initialized during the startup of Spring Boot and interferes between the tests.

Is it possible to use Spring Boot integration tests with Hibernate Search in a way so that locks are cleanly released between tests? Alternativly I'm looking for a way to disable the Hibernate Search indexing for all integration tests not making use of Hibernate Search

I also tried already the property near-real-time and different lock factories as native, simple and single without luck.

Answer 1

First: do not use exclusive_index_use unless you are a Lucene guru. It is dangerous and probably will not behave as you want.

Now that we got that out of the way... As far as I understand, you are trying to execute integration tests in parallel on the same machine. This means integration tests will probably compete for access to the exact same index, and will write to the same index. This could lead to unpredictable results if your tests perform conflicting writes (one test erasing a document added by another test, before that test has completed).

If you really need to perform tests in parallel, I would recommend to execute each test in an isolated environment:

• Dedicated DB, or at least dedicated DB schema
• Dedicated Lucene index
• etc.

In the case of Hibernate Search, you will have to find a way to use a different physical index in each test execution.

There are two ways to do that:

1. Just for tests, do not store the indexes on the filesystem, but directly in the heap, by setting hibernate.search.backend.directory.type to local-heap (Hibernate Search 6+) or hibernate.search.default.directory_provider to local-heap (Hibernate Search 5 and below).

It's super easy to implement, but there are a few disadvantages you should be aware of:

• your testing environment will not be exactly the same as your production environment anymore
• indexes will be lost after the tests have finished executing, which may make post-mortem debugging challenging (you won't be able to use Luke to inspect the state of the indexes anymore)
• if your integration tests store a lot of content in the index, you might get an OutOfMemoryError.

2. If the disadvantages of solution 1 are too much for you, you can continue using the filesystem to store the indexes, but use a different configuration for each test execution, setting the index base path (hibernate.search.backend.directory.root for Hibernate Search 6+, or hibernate.search.default.indexBase for Hibernate Search 5 and below) to some unique path for each test execution. You will have to find how to do that in Spring, but I would be suprised to learn it's not possible. Maybe Spring allows you to use interpolation in the properties, something like hibernate.search.backend.directory.root = /tmp/it/#{testName}?

See the documentation about directory configuration (here for Hibernate Search 6+, or here for Hibernate Search 5 for more information on how to configure index storage.