Trying to avoid search timeouts with the new Java elastic client

Question

I'm trying to avoid timeouts when running some queries in my local elastic 8.7.0 server. While most of them run without issues, some of them time out for whatever reason, throwing the following exception:

java.net.SocketTimeoutException: 30.000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:915)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:300)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
        at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
        at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1833)
        at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1850)
        at sid.EntitySearcher.ElasticEntitySearcher.executeElasticQuery(ElasticEntitySearcher.java:195)
        at sid.EntitySearcher.ElasticEntitySearcher.getScoredElasticResults(ElasticEntitySearcher.java:242)
        at sid.EntitySearcher.ElasticEntitySearcher.scoredSearch(ElasticEntitySearcher.java:54)
        at sid.Evaluation.Evaluator.runQueriesAndSaveAsTRECResultsFile(Evaluator.java:81)
        at sid.MetricsBulkTesting.testSingleEngine(MetricsBulkTesting.java:308)
        at sid.MetricsBulkTesting.testSingleEngines(MetricsBulkTesting.java:430)
        at sid.MetricsBulkTesting.main(MetricsBulkTesting.java:84)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:578)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:95)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: java.net.SocketTimeoutException: 30.000 milliseconds timeout on connection http-outgoing-3 [ACTIVE]
        at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
        at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:98)
        at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:40)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.base/java.lang.Thread.run(Thread.java:1589)

I have tried to tweak this timeout value, but nothing I try seems to work:

I have built the client like this, tweaking the keepAlive values as per this Github issue, although it has no effect:

RestClient restClient = RestClient
                .builder(new HttpHost(host, port, "https"))
                .setHttpClientConfigCallback(hc -> hc
                        .setSSLContext(sslContext)
                        .setDefaultCredentialsProvider(credsProv)
                        // Avoid timeout problems
                        // https://github.com/elastic/elasticsearch/issues/65213
                        .setKeepAliveStrategy((response, context) -> 300000/* 5 minutes*/)
                        .setDefaultIOReactorConfig(IOReactorConfig.custom().setSoKeepAlive(true).build())
                )
                .build();

        // Create the transport and the API client
        ElasticsearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());

Also, when searching, I have tried to tweak the search request's timeout value, but it also has no effect at all:

SearchResponse<EntityDocument> response = connector.getClient().search(s -> s
                            .index(connector.getIndexName())
                            .timeout("180s")
                            .query(q -> q
                                    .match(t -> t
                                            ...query stuff
                                    )
                            ),
                    EntityDocument.class
            );

I also reopen a fresh connection to the elastic server when timing out a search for the first time, with, again, no effect at all.

Is the issue somewhere else? Did I miss a parameter somewhere?

Another issue would be the searches themselves, which are suspiciosly slow. When doing bulk indexes of thousands of documents per second on the same connection for ~5 hours, there are no issues at all.

EDIT: Solved... the underlying culprit was querying n-gram subfields...now the queries take milliseconds, so it's not needed anymore

queries above 30s is normal? maybe it would be better to refactor the queries and have better performance. — rabbitbr, Apr 30 '23 at 17:45
@rabbitbr It's also part of what I am concerned about. It's only a phrase query (the longest being 13 terms), thrown at three text fields, each of them having a ngrams and keyword subfield. I have 22M documents which don't really have much text in them. — Lightsong, Apr 30 '23 at 17:57
@rabbitbr Thanks! Will check it. I managed to somewhat avoid it though....turns out the query times were NOT normal at all...the ngrams subfield made them jump from like twenty seconds to milliseconds... — Lightsong, Apr 30 '23 at 20:56

Trying to avoid search timeouts with the new Java elastic client

0 Answers0