Let's Encrypt + Hitch: Android OkHttp java.security.cert.CertPathValidatorException: Response is unreliable: its validity interval is out-of-date

Question

Hi and thank you for helping!

Very infrequently on our Android application, we get errors that start as java.security.cert.CertificateException: Chain validation failed with then traces to the exception of Response is unreliable: its validity interval is out-of-date. This impacts API calls and media playback (as ExoPlayer 2 uses okhttp).

We manage SSL transactions via Hitch, and acquire our certificates from Let's Encrypt. What's odd is that when this issue happens, the certificates still work fine on web, iOS tvOS, and Roku, but not on Android.

The error is intermittent, and in a matter of 24-48 hours, it seems this exception no longer happens. On okhttp I see StackOverflow articles where users suggest adding custom TrustManager class into okhttp that accepts all certificates, but I see that as insecure.

My question are how can I properly log what validity interval it's receiving, vs what it expects to have, and how can I tweak the settings at the Java level to trust beyond that window so that this exception doesn't happen?

For context, here is a stack trace affecting playback.


2019-10-21 16:42:51.883 31064-32727/com.sbs.lamusica.debug E/ExoPlayerImplInternal: Source error.
    com.google.android.exoplayer2.upstream.HttpDataSource$HttpDataSourceException: Unable to connect to https://interstitials.lamusica.com/lamusica/black-video.mp4
        at com.google.android.exoplayer2.upstream.DefaultHttpDataSource.open(DefaultHttpDataSource.java:194)
        at com.google.android.exoplayer2.upstream.DefaultDataSource.open(DefaultDataSource.java:147)
        at com.google.android.exoplayer2.source.ExtractorMediaPeriod$ExtractingLoadable.load(ExtractorMediaPeriod.java:841)
        at com.google.android.exoplayer2.upstream.Loader$LoadTask.run(Loader.java:308)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:764)
     Caused by: javax.net.ssl.SSLHandshakeException: Chain validation failed
        at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:236)
        at com.android.okhttp.internal.io.RealConnection.connectTls(RealConnection.java:1480)
        at com.android.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:1424)
        at com.android.okhttp.internal.io.RealConnection.connect(RealConnection.java:1368)
        at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:219)
        at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:142)
        at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:104)
        at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:392)
        at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:325)
        at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:470)
        at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:127)
        at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.connect(DelegatingHttpsURLConnection.java:89)
        at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:26)
        at com.google.android.exoplayer2.upstream.DefaultHttpDataSource.makeConnection(DefaultHttpDataSource.java:429)
        at com.google.android.exoplayer2.upstream.DefaultHttpDataSource.makeConnection(DefaultHttpDataSource.java:350)
        at com.google.android.exoplayer2.upstream.DefaultHttpDataSource.open(DefaultHttpDataSource.java:192)
        at com.google.android.exoplayer2.upstream.DefaultDataSource.open(DefaultDataSource.java:147) 
        at com.google.android.exoplayer2.source.ExtractorMediaPeriod$ExtractingLoadable.load(ExtractorMediaPeriod.java:841) 
        at com.google.android.exoplayer2.upstream.Loader$LoadTask.run(Loader.java:308) 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) 
        at java.lang.Thread.run(Thread.java:764) 
     Caused by: java.security.cert.CertificateException: Chain validation failed
        at com.android.org.conscrypt.TrustManagerImpl.verifyChain(TrustManagerImpl.java:719)
        at com.android.org.conscrypt.TrustManagerImpl.checkTrustedRecursive(TrustManagerImpl.java:543)
        at com.android.org.conscrypt.TrustManagerImpl.checkTrustedRecursive(TrustManagerImpl.java:564)
        at com.android.org.conscrypt.TrustManagerImpl.checkTrustedRecursive(TrustManagerImpl.java:632)
        at com.android.org.conscrypt.TrustManagerImpl.checkTrusted(TrustManagerImpl.java:499)
        at com.android.org.conscrypt.TrustManagerImpl.checkTrusted(TrustManagerImpl.java:422)
        at com.android.org.conscrypt.TrustManagerImpl.getTrustedChainForServer(TrustManagerImpl.java:343)
        at android.security.net.config.NetworkSecurityTrustManager.checkServerTrusted(NetworkSecurityTrustManager.java:94)
        at android.security.net.config.RootTrustManager.checkServerTrusted(RootTrustManager.java:88)
        at com.android.org.conscrypt.Platform.checkServerTrusted(Platform.java:208)
        at com.android.org.conscrypt.ConscryptFileDescriptorSocket.verifyCertificateChain(ConscryptFileDescriptorSocket.java:426)
        at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method)
        at com.android.org.conscrypt.NativeSsl.doHandshake(NativeSsl.java:383)
        at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:231)
        at com.android.okhttp.internal.io.RealConnection.connectTls(RealConnection.java:1480) 
        at com.android.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:1424) 
        at com.android.okhttp.internal.io.RealConnection.connect(RealConnection.java:1368) 
        at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:219) 
        at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:142) 
        at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:104) 
        at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:392) 
        at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:325) 
        at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:470) 
        at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:127) 
        at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.connect(DelegatingHttpsURLConnection.java:89) 
        at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:26) 
        at com.google.android.exoplayer2.upstream.DefaultHttpDataSource.makeConnection(DefaultHttpDataSource.java:429) 
        at com.google.android.exoplayer2.upstream.DefaultHttpDataSource.makeConnection(DefaultHttpDataSource.java:350) 
        at com.google.android.exoplayer2.upstream.DefaultHttpDataSource.open(DefaultHttpDataSource.java:192) 
        at com.google.android.exoplayer2.upstream.DefaultDataSource.open(DefaultDataSource.java:147) 
        at com.google.android.exoplayer2.source.ExtractorMediaPeriod$ExtractingLoadable.load(ExtractorMediaPeriod.java:841) 
        at com.google.android.exoplayer2.upstream.Loader$LoadTask.run(Loader.java:308) 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) 
        at java.lang.Thread.run(Thread.java:764) 
     Caused by: java.security.cert.CertPathValidatorException: Response is unreliable: its validity interval is out-of-date
2019-10-21 16:42:51.886 31064-32727/com.sbs.lamusica.debug E/ExoPlayerImplInternal:     at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:222)
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140)
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)
        at com.android.org.conscrypt.DelegatingCertPathValidator.engineValidate(DelegatingCertPathValidator.java:44)
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:301)
        at com.android.org.conscrypt.TrustManagerImpl.verifyChain(TrustManagerImpl.java:715)
            ... 34 more
     Caused by: java.security.cert.CertPathValidatorException: Response is unreliable: its validity interval is out-of-date
        at sun.security.provider.certpath.OCSPResponse.verify(OCSPResponse.java:619)
        at sun.security.provider.certpath.RevocationChecker.checkOCSP(RevocationChecker.java:709)
        at sun.security.provider.certpath.RevocationChecker.check(RevocationChecker.java:363)
        at sun.security.provider.certpath.RevocationChecker.check(RevocationChecker.java:337)
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
            ... 40 more
        Suppressed: java.security.cert.CertPathValidatorException: Could not determine revocation status
        at sun.security.provider.certpath.RevocationChecker.buildToNewKey(RevocationChecker.java:1092)
        at sun.security.provider.certpath.RevocationChecker.verifyWithSeparateSigningKey(RevocationChecker.java:910)
        at sun.security.provider.certpath.RevocationChecker.checkCRLs(RevocationChecker.java:577)
        at sun.security.provider.certpath.RevocationChecker.checkCRLs(RevocationChecker.java:465)
        at sun.security.provider.certpath.RevocationChecker.check(RevocationChecker.java:394)

Here is another example stacktrace affecting an API call.

javax.net.ssl.SSLHandshakeException: Chain validation failed
    at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:236)
    at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:320)
    at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:284)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:169)
    at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:264)
    at okhttp3.RealCall.execute(RealCall.java:93)
    at com.sbs.lamusica.ui.fragment.home.HomePagerFragment.run(HomePagerFragment.java:200)
    at com.sbs.lamusica.ui.fragment.home.HomePagerFragment$FetchLocationsList.doInBackground(HomePagerFragment.java:706)
    at com.sbs.lamusica.ui.fragment.home.HomePagerFragment$FetchLocationsList.doInBackground(HomePagerFragment.java:693)
    at android.os.AsyncTask$2.call(AsyncTask.java:333)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:764)
Caused by: java.security.cert.CertificateException: Chain validation failed
    at com.android.org.conscrypt.TrustManagerImpl.verifyChain(TrustManagerImpl.java:719)
    at com.android.org.conscrypt.TrustManagerImpl.checkTrustedRecursive(TrustManagerImpl.java:543)
    at com.android.org.conscrypt.TrustManagerImpl.checkTrustedRecursive(TrustManagerImpl.java:564)
    at com.android.org.conscrypt.TrustManagerImpl.checkTrustedRecursive(TrustManagerImpl.java:632)
    at com.android.org.conscrypt.TrustManagerImpl.checkTrusted(TrustManagerImpl.java:499)
    at com.android.org.conscrypt.TrustManagerImpl.checkTrusted(TrustManagerImpl.java:422)
    at com.android.org.conscrypt.TrustManagerImpl.getTrustedChainForServer(TrustManagerImpl.java:343)
    at android.security.net.config.NetworkSecurityTrustManager.checkServerTrusted(NetworkSecurityTrustManager.java:94)
    at android.security.net.config.RootTrustManager.checkServerTrusted(RootTrustManager.java:88)
    at com.android.org.conscrypt.Platform.checkServerTrusted(Platform.java:208)
    at com.android.org.conscrypt.ConscryptFileDescriptorSocket.verifyCertificateChain(ConscryptFileDescriptorSocket.java:426)
    at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method)
    at com.android.org.conscrypt.NativeSsl.doHandshake(NativeSsl.java:383)
    at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:231)
    ... 28 more
Caused by: java.security.cert.CertPathValidatorException: Response is unreliable: its validity interval is out-of-date
    at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
    at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:222)
    at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:140)
    at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:79)
    at com.android.org.conscrypt.DelegatingCertPathValidator.engineValidate(DelegatingCertPathValidator.java:44)
    at java.security.cert.CertPathValidator.validate(CertPathValidator.java:301)
    at com.android.org.conscrypt.TrustManagerImpl.verifyChain(TrustManagerImpl.java:715)
    ... 41 more
Caused by: java.security.cert.CertPathValidatorException: Response is unreliable: its validity interval is out-of-date
    at sun.security.provider.certpath.OCSPResponse.verify(OCSPResponse.java:619)
    at sun.security.provider.certpath.RevocationChecker.checkOCSP(RevocationChecker.java:709)
    at sun.security.provider.certpath.RevocationChecker.check(RevocationChecker.java:363)
    at sun.security.provider.certpath.RevocationChecker.check(RevocationChecker.java:337)
    at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
    ... 47 more
    Suppressed: java.security.cert.CertPathValidatorException: Could not determine revocation status
    at sun.security.provider.certpath.RevocationChecker.buildToNewKey(RevocationChecker.java:1092)
    at sun.security.provider.certpath.RevocationChecker.verifyWithSeparateSigningKey(RevocationChecker.java:910)
    at sun.security.provider.certpath.RevocationChecker.checkCRLs(RevocationChecker.java:577)
    at sun.security.provider.certpath.RevocationChecker.checkCRLs(RevocationChecker.java:465)
    at sun.security.provider.certpath.RevocationChecker.check(RevocationChecker.java:394)
        ... 49 more
2019-07-24 23:45:18.793 1473-3146/? V/WindowManager: Relayout Window{87d24e5 u0 com.sbs.lamusica.deb

Looks the same as https://stackoverflow.com/a/54785688/115145 -- are you using OCSP stapling? — CommonsWare, Apr 21 '20 at 21:05
Any chance the android device has an incorrect time https://stackoverflow.com/a/57180523/1542667 — Yuri Schimke, Apr 25 '20 at 12:30

score 0 · Answer 1 · answered May 17 '20 at 07:57

Thanks for everyone's help!

After countless searching, I finally learned that hitch actually automatically manages the retrieval of OCSP staples and it can be configured to attach them.

After our app resurfaced with this issue just now at this obscene hour, I applied the following changes on my RedHat server:

Create a /var/lib/hitch-ocsp directory
Set the ownership of /var/lib/hitch-ocsp to the hitch user
Added the following two options to the /etc/hitch/hitch.conf

ocsp-dir = "/var/lib/hitch-ocsp"
ocsp-verify-staple = on

Ran service hitch restart to rerun hitch and now I started seeing OCSP stapling on my SSL responses whenever I ran it against https://entrust.ssllabs.com/analyze.html

So far immediately after the change I am now seeing our Android app properly trust these requests, but I'll post here if the issue resurfaces.

score 0 · Answer 2 · answered Feb 01 '23 at 06:22

0

In my case, it was enough to just set the proper system time on the device. ('Settings' > 'System' > 'Date & Time' > 'Automatic date & time').

answered Feb 01 '23 at 06:22

Bhavesh Butani

281
2
18

Let's Encrypt + Hitch: Android OkHttp java.security.cert.CertPathValidatorException: Response is unreliable: its validity interval is out-of-date

2 Answers2