I have an application that uses openjdk-11.0.1 java.net.HttpClient to make a multipart POST request to a REST(ish) endpoint. That endpoint blocks the response while it does whatever work, and when the work is done, it returns the response. The nature of this endpoint is such that a response can take anywhere from 1s to 1d, so this connection can be held for a very long time.
So anyway, recently I've started to encounter the below error, even though the process on the target server still seems to be executing just fine. Does anyone have any ideas about possible causes to this error?
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: HTTP/1.1 header parser received no bytes
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at com.company.perf.rdp.profile.DatasetProfiler.toJson(DatasetProfiler.java:190)
... 9 common frames omitted
Caused by: java.io.IOException: HTTP/1.1 header parser received no bytes
at java.net.http/jdk.internal.net.http.common.Utils.wrapWithExtraDetail(Utils.java:293)
at java.net.http/jdk.internal.net.http.Http1Response$HeadersReader.onReadError(Http1Response.java:657)
at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.checkForErrors(Http1AsyncReceiver.java:297)
at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:263)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
at java.net.http/jdk.internal.net.http.HttpClientImpl$DelegatingExecutor.execute(HttpClientImpl.java:153)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:273)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:242)
at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.onReadError(Http1AsyncReceiver.java:506)
at java.net.http/jdk.internal.net.http.Http1AsyncReceiver$Http1TubeSubscriber.onComplete(Http1AsyncReceiver.java:591)
at java.net.http/jdk.internal.net.http.common.SSLTube$DelegateWrapper.onComplete(SSLTube.java:268)
at java.net.http/jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.complete(SSLTube.java:411)
at java.net.http/jdk.internal.net.http.common.SSLTube$SSLSubscriberWrapper.onComplete(SSLTube.java:540)
at java.net.http/jdk.internal.net.http.common.SubscriberWrapper.checkCompletion(SubscriberWrapper.java:443)
at java.net.http/jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run1(SubscriberWrapper.java:322)
at java.net.http/jdk.internal.net.http.common.SubscriberWrapper$DownstreamPusher.run(SubscriberWrapper.java:261)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:271)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(SequentialScheduler.java:224)
at java.net.http/jdk.internal.net.http.common.SubscriberWrapper.outgoing(SubscriberWrapper.java:234)
at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:467)
at java.net.http/jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:263)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
... 3 common frames omitted
Caused by: java.io.EOFException: EOF reached while reading
... 21 common frames omitted
Here's the creation of my HttpClient:
default HttpClient create() {
final SSLContextBuilder sslContextBuilder;
try {
sslContextBuilder = new SSLContextBuilder().loadTrustMaterial(TrustSelfSignedStrategy.INSTANCE);
} catch (NoSuchAlgorithmException | KeyStoreException e) {
throw new RuntimeException("Unable to build SSLContext", e);
}
// PREVENTS HOST VALIDATION
final Properties props = System.getProperties();
props.setProperty("jdk.internal.httpclient.disableHostnameVerification", Boolean.TRUE.toString());
// SHOULD PREVENT HOST VALIDATION
final SSLParameters sslParams = new SSLParameters();
sslParams.setEndpointIdentificationAlgorithm(null);
try {
final SSLContext sslContext = sslContextBuilder.build();
ignoreExpiredCerts(sslContext);
return HttpClient.newBuilder().version(Version.HTTP_1_1).sslContext(sslContext).sslParameters(sslParams)
.build();
} catch (KeyManagementException | NoSuchAlgorithmException e) {
throw new RuntimeException("Unable to build HttpClient", e);
}
}
private void ignoreExpiredCerts(final SSLContext sslContext) throws KeyManagementException {
TrustManagerFactory tmf;
try {
tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
} catch (final NoSuchAlgorithmException e) {
throw new RuntimeException("Unable to build HttpClient", e);
}
try {
tmf.init((KeyStore) null);
} catch (final KeyStoreException e) {
throw new RuntimeException("Unable to build HttpClient", e);
}
final TrustManager[] trustManagers = tmf.getTrustManagers();
final X509TrustManager origTrustmanager = (X509TrustManager) trustManagers[0];
final AtomicBoolean logged = new AtomicBoolean(false);
final TrustManager[] wrappedTrustManagers = new TrustManager[] { new X509TrustManager() {
@Override
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return origTrustmanager.getAcceptedIssuers();
}
@Override
public void checkClientTrusted(final X509Certificate[] certs, final String authType)
throws CertificateException {
origTrustmanager.checkClientTrusted(certs, authType);
}
@Override
public void checkServerTrusted(final X509Certificate[] certs, final String authType)
throws CertificateException {
try {
origTrustmanager.checkServerTrusted(certs, authType);
} catch (final CertificateExpiredException e) {
if (!logged.get()) {
LOGGER.warn("Server certificate expired", e);
logged.set(true);
}
} catch (final Exception e) {
if (e.getCause() != null && e.getCause().getCause() != null
&& e.getCause().getCause() instanceof CertificateExpiredException) {
if (!logged.get()) {
LOGGER.warn("Server certificate expired", e.getCause().getCause());
logged.set(true);
}
} else {
throw e;
}
}
}
} };
sslContext.init(null, wrappedTrustManagers, null);
}
update:
i did not find an answer to this. upgrading to newest java version helps (8 vs 14 changes a lot under the hood for java httpclient).
additionally, this response often is generated by the downstream server, not the client. meaning that the server to which i was connecting closed the connection. we think this is because the connection is detected as "idle" while transferring large volumes of data.
for older java httpclients, turning off the keepalive seems to help this issue as well. before instantiating your httpclient, make these calls:
// ALLOWS CONNECTION CLOSE HEADER
props.setProperty("jdk.httpclient.allowRestrictedHeaders", EnumSet.allOf(HttpHeaders.class).stream().filter(HttpHeaders::isRestricted).map(HttpHeaders::getName).map(StringUtils::lowerCase).collect(Collectors.joining(",")));
// TURN IT OFF
props.setProperty("jdk.httpclient.keepalive.timeout", "0");
on each request, i close the connection:
// CLOSE SOCKET AFTER EACH REQUEST
requestBuilder.header(HttpHeaders.CONNECTION.getName(), "close");
the referenced HttpHeaders class:
package com.company.perf.api.http;
import org.apache.commons.lang3.StringUtils;
import com.company.api.http.client.IMakeHttpRequests;
public enum HttpHeaders {
ACCEPT("Accept"),
ACCEPT_CHARSET("Accept-Charset"),
ACCEPT_ENCODING("Accept-Encoding"),
ACCEPT_LANGUAGE("Accept-Language"),
ALLOW("Allow"),
AUTHORIZATION("Authorization"),
BOUNDARY("boundary"),
CACHE_CONTROL("Cache-Control"),
CONNECTION("Connection", true),
CONTENT_DISPOSITION("Content-Disposition"),
CONTENT_ENCODING("Content-Encoding"),
CONTENT_ID("Content-ID"),
CONTENT_LANGUAGE("Content-Language"),
CONTENT_LENGTH("Content-Length", true),
CONTENT_LOCATION("Content-Location"),
CONTENT_TYPE("Content-Type"),
COOKIE("Cookie"),
DATE("Date"),
ETAG("ETag"),
EXPECT("Expect", true),
EXPIRES("Expires"),
HOST("Host", true),
IF_MATCH("If-Match"),
IF_MODIFIED_SINCE("If-Modified-Since"),
IF_NONE_MATCH("If-None-Match"),
IF_UNMODIFIED_SINCE("If-Unmodified-Since"),
LAST_MODIFIED("Last-Modified"),
LINK("Link"),
LOCATION("Location"),
RETRY_AFTER("Retry-After"),
SET_COOKIE("Set-Cookie"),
UPGRADE("Upgrade", true),
USER_AGENT("User-Agent"),
VARY("Vary"),
WWW_AUTHENTICATE("WWW-Authenticate");
private final String name;
private final boolean restricted;
private HttpHeaders(final String name, final boolean restricted) {
assert StringUtils.isNotBlank(name) : "name cannot be blank";
this.name = name;
this.restricted = restricted;
}
private HttpHeaders(final String name) {
this(name, false);
}
public String getName() {
return this.name;
}
/**
* @return whether or not this header's use is restricted by the
* {@link IMakeHttpRequests Java HTTP Client}
*/
public boolean isRestricted() {
return this.restricted;
}
}
