Query with Hibernate Search and get all items without paging

Question

I'm using Hibernate Search in my Spring Boot project. The most of the time it's a perfect solution for my needs except in one case when I have to get all items selected by full text search query.

Is there a chance to avoid Hibernate Search / Lucene paging while using Hibernate Search? Something similar to Pageable.unpaged()? I don't want to set page size to Integer.MAX_VALUE as it's kind of hack. But if there is no better solution, I'll have to use it that way.

yrodiere · Accepted Answer · 2022-08-11T07:51:24.427

You generally shouldn't do fetch all hits in a single request, because there could be millions of hits and loading them all is likely to cause memory issues.

If you really want to do this regardless, you can fetch all results simply by not setting a limit on your query:

Hibernate Search 6:

Session session = /* ... */;
List<MyEntity> = Search.session( session ).search( MyEntity.class )
        .where( f -> /* ... */ )
        .fetchAllHits();

Hibernate Search 5:

Session session = /* ... */;
FullTextSession fullTextSession = Search.getFullTextSession( session );
Query luceneQuery = /* ... */;
FullTextQuery ftQuery = fullTextSession.createFullTextQuery( luceneQuery, MyEntity.class );
List<MyEntity> hits = ftQuery.list();

But in many cases it won't work or will perform poorly because the hits take too much memory.

Ideally, you should use scrolling instead.

Here is an example with Hibernate Search 6:

SearchSession searchSession = /*...*/;
try ( SearchScroll<MyEntity> scroll = Search.session( session )
        .search( MyEntity.class )
        .where( f -> f.matchAll() )
        .scroll( 20 ) ) { 
    for ( SearchScrollResult<MyEntity> chunk = scroll.next(); 
            chunk.hasHits(); chunk = scroll.next() ) { 
        for ( MyEntity entity : chunk.hits() ) { 
            // ... do things with the entity ...
        }

        totalHitCount = chunk.total().hitCount(); 

        entityManager.flush();
        entityManager.clear();
    }
}

Hibernate Search 5:

FullTextQuery ftQuery = /*...*/;
ScrollableResults scroll = ftQuery.scroll();
scroll.beforeFirst();
int i = 0;
while ( scroll.next() ) {
    ++i;
    MyEntity entity = scroll.get()[0];

    // ... do things with the entity ...

    // Flush/clear the session periodically to free memory
    if ( i % 100 == 0 ) {
        // Only necessary if you changed entities
        session.flush();
        // Only necessary if you changed indexed entities
        // Also, only necessary in Hibernate Search 5
        fullTextSession.flushToIndexes();

        session.clear();
    }
}

Thank you! I use Paging for all my queries by setting max result and 1st item but this one specific case would cover a very limited set of max 100 items as they are simple educational articles of few text lines. So there is no chance to ever get more than 100 or 120 and that should not be a problem to process in one fetch. — horvoje, Jul 17 '20 at 23:08
In the other hand, the scroll patthern you provided looks like good solution too as I'm building a list of links to the articles - instantiate a list for URLs, run a query and scroll over all articles and put their URL into the list. — horvoje, Jul 17 '20 at 23:14
@horvoje You might want to use projections instead, to load the URL from the data stored in the index; that will save you a database access. More information: https://docs.jboss.org/hibernate/search/5.11/reference/en-US/html_single/#projections — yrodiere, Jul 20 '20 at 06:17

Query with Hibernate Search and get all items without paging

1 Answers1