Perform JPA search in rows with given Ids

Question

I am working on making the search in my application more efficient. The use case that I am trying to solve is that given a set of Ids, perform search only in those given rows. The problem is that the size of this ids is more than 1024. When I use BooleanQuery.setMaxClauseCount(Ids.size() I don't get the tooManyClauses Exception but it doesn't return any result because of timeOut.

Ideally I would like to do this:

public ResponseEntity search(
            @RequestParam(Param.SEARCH) String query,
            @RequestParam(value = Param.PAGE, defaultValue = Pagination.DEFAULT_PAGE) int page,
            @RequestParam(value = Param.SIZE, defaultValue = Pagination.DEFAULT_SIZE) int size)
            throws Exception {
        log.info("Started searching for query : " + query);

        final Set<Long> accessibleIds = projectsServiceUtils
                .filterIdsAccessibleByLoggedInPerson(null);

        final FullTextQuery fullTextQuery = searchHelper
                .getPrefixSearchQuery(Project.class, SearchFields.PROJECT_BOOST_MAP, query, accessibleRadarIds);

        final List<Project> projects = fullTextQuery.setFirstResult(page * size)
                .setMaxResults(size).getResultList();

        return ResponseEntity.ok()
                .header(Pagination.PAGINATION_PAGE, String.valueOf(page))
                .header(Pagination.PAGINATION_SIZE, String.valueOf(size))
                .header(Pagination.PAGINATION_COUNT, Long.toString(fullTextQuery.getResultSize()))
                .body(projectSearchObjectWriter.writeValueAsString(projects));
    }

With this as my getPrefixSearchQuery Method:

public <T> FullTextQuery getPrefixSearchQuery(
            Class<T> typeClass, Map<String, Float> boostMap, String searchTerms, Set<Long> ids) {
        FullTextEntityManager fullTextEntityManager = Search
                .getFullTextEntityManager(entityManager);
        QueryBuilder qb = fullTextEntityManager
                .getSearchFactory()
                .buildQueryBuilder()
                .forEntity(typeClass).get();

        BooleanQuery.Builder luceneQuery = new BooleanQuery.Builder();

        String[] tokens = searchTerms.split("\\s+");

        for (String token : tokens) {
            if (!StopAnalyzer.ENGLISH_STOP_WORDS_SET.contains(token) || tokens.length == 1) {
                // If search term contains only digits then search substring (possibly phone number)
                final String matcher = token.toLowerCase() + "*";

                final WildcardContext wildcardContext = qb.keyword().wildcard();
                TermMatchingContext termMatchingContext = null;
                for (String field : boostMap.keySet()) {
                    if (termMatchingContext != null) {
                        termMatchingContext = termMatchingContext.andField(field).boostedTo(boostMap.get(field));
                    } else {
                        termMatchingContext = wildcardContext.onField(field).boostedTo(boostMap.get(field));
                    }
                }
                final Query subQuery = termMatchingContext.matching(matcher).createQuery();
                luceneQuery.add(subQuery, BooleanClause.Occur.MUST);
            }
        }
        // NEW CODE TO SUPPORT FILTERING
        if (ids != null) {
            BooleanQuery.setMaxClauseCount(ids.size() + (tokens.length*boostMap.size()));
            TermMatchingContext termMatchingContext2 = qb.keyword().wildcard().onField("id");

            for (Long id : ids) {
                luceneQuery.add(termMatchingContext2.matching(id).createQuery(), BooleanClause.Occur.FILTER);
            }
        }

        FullTextQuery jpaQuery = fullTextEntityManager
                .createFullTextQuery(luceneQuery.build(), typeClass);

        return jpaQuery;
    }

Since I am not getting any results with the above config I have to filter my results after getting the query result. This causes more issues since I have to make sure the size of the result is equal to the size passed, so instead of filtering through the first page of results, I have to filter through the entire result set and then paginate it to get the result of desired size. This is my very inefficient work around right now:

public ResponseEntity search(
            @RequestParam(Param.SEARCH) String query,
            @RequestParam(value = Param.PAGE, defaultValue = Pagination.DEFAULT_PAGE) int page,
            @RequestParam(value = Param.SIZE, defaultValue = Pagination.DEFAULT_SIZE) int size,
            @RequestParam(value = Param.SORT, defaultValue = SortDefault.BY_NAME) String sort)
            throws IOException, IdmsException {
        log.info("Started searching for query : " + query);

        final FullTextQuery fullTextQuery = searchHelper
                .getPrefixSearchQuery(Project.class, SearchFields.PROJECT, query);
        final List<Project> projects = fullTextQuery.getResultList();

        final ImmutableSet<Long> Ids = projects
                .stream()
                .map(Project::getId)
                .collect(collectingAndThen(Collectors.toSet(),
                        ImmutableSet::copyOf));

        final Set<Long> accessibleIds = projectsServiceUtils
                .filterIdsAccessibleByLoggedInPerson(Ids);

        PageRequest pageRequest = new PageRequest(
                page, size);

        final Page<Project> projectsFiltered = projectRepository.findByIdIn(
                accessibleIds, pageRequest);

        return ResponseEntity.ok()
                .header(Pagination.PAGINATION_PAGE, String.valueOf(page))
                .header(Pagination.PAGINATION_SIZE, String.valueOf(size))
                .header(Pagination.PAGINATION_COUNT, String.valueOf(accessibleIds.size()))
                .body(projectSearchObjectWriter.writeValueAsString(projectsFiltered.getContent()));
    }

Is there any way I can perform search in rows with the given Id and get it paginated?

score 0 · Answer 1 · answered Apr 17 '19 at 11:39

    if (ids != null) {
       BooleanQuery.setMaxClauseCount(ids.size() + (tokens.length*boostMap.size()));
       TermMatchingContext termMatchingContext2 = qb.keyword().wildcard().onField("id");

       for (Long id : ids) {
           luceneQuery.add(termMatchingContext2.matching(id).createQuery(), BooleanClause.Occur.FILTER);
       }
   }

You're using a wildcard query, but you do not need to. Depending on the size of your ID list, this might be the reason you're getting a timeout.

Try this instead:

     if (ids != null) {
        BooleanQuery.setMaxClauseCount(ids.size() + (tokens.length*boostMap.size()));

        for (Long id : ids) {
            luceneQuery.add(qb.keyword().onField("id").matching(id).createQuery(), BooleanClause.Occur.FILTER);
        }
    }

Also, you do not need to set the max clause count every time you perform a query. You really shouldn't, actually, since multiple queries may be executed in parallel.

Just execute something like this when your application starts:

BooleanQuery.setMaxClauseCount(<some large number>);

Perform JPA search in rows with given Ids

1 Answers1