Error Connecting to Secret Manager - TLS ALPN negotiation failed with protocols: [h2]

Question

I am trying to connect to Google Secret Manager from my linux server and that's when I am getting following error.

Java Version:

java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

Error

com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:67)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
        at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
        at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1387)
        at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:398)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1015)
        at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:672)
        at com.google.common.util.concurrent.ForwardingListenableFuture.addListener(ForwardingListenableFuture.java:45)
        at com.google.api.core.ApiFutureToListenableFuture.addListener(ApiFutureToListenableFuture.java:52)
        at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1368)
        at com.google.api.core.ApiFutures.addCallback(ApiFutures.java:63)
        at com.google.api.gax.grpc.GrpcExceptionCallable.futureCall(GrpcExceptionCallable.java:67)
        at com.google.api.gax.rpc.AttemptCallable.call(AttemptCallable.java:87)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
                at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
                at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
                at com.google.cloud.secretmanager.v1.SecretManagerServiceClient.accessSecretVersion(SecretManagerServiceClient.java:1190)
                at com.google.cloud.secretmanager.v1.SecretManagerServiceClient.accessSecretVersion(SecretManagerServiceClient.java:1130)
                at com.economical.services.secretmanager.SecretManager.pullSecret(SecretManager.java:16)
                at com.economical.App.main(App.java:12)
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE
        at io.grpc.Status.asRuntimeException(Status.java:535)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:534)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:455)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        ... 3 more
Caused by: java.lang.RuntimeException: TLS ALPN negotiation failed with protocols: [h2]
        at io.grpc.okhttp.OkHttpProtocolNegotiator.negotiate(OkHttpProtocolNegotiator.java:104)
        at io.grpc.okhttp.OkHttpTlsUpgrader.upgrade(OkHttpTlsUpgrader.java:63)
        at io.grpc.okhttp.OkHttpClientTransport$3.run(OkHttpClientTransport.java:548)
        ... 4 more

POM.XML

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.economical</groupId>
    <artifactId>HDFSReplicator</artifactId>
    <version>1.0</version>
    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-distcp</artifactId>
            <version>3.0.0</version>
            <scope>provided</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>3.0.0</version>
            <scope>provided</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-core -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>3.0.0</version>
            <scope>provided</scope>
        </dependency>


        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
            <version>3.0.0</version>
            <scope>provided</scope>
        </dependency>

        <!--        <dependency>-->
        <!--            <groupId>log4j</groupId>-->
        <!--            <artifactId>log4j</artifactId>-->
        <!--            <version>1.2.17</version>-->
        <!--        </dependency>-->

        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.16.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.16.0</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java -->
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>8.0.30</version>
        </dependency>

        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-secretmanager</artifactId>
            <version>2.1.3</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java -->
        <dependency>
            <groupId>com.google.protobuf</groupId>
            <artifactId>protobuf-java</artifactId>
            <version>3.20.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.google.cloud/google-cloud-core -->
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-core</artifactId>
            <version>2.8.6</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.google.guava/guava -->
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>23.6-jre</version>
        </dependency>

<!--        <dependency>-->
<!--            <groupId>org.apache.httpcomponents</groupId>-->
<!--            <artifactId>httpcore</artifactId>-->
<!--            <version>4.4.8</version>-->
<!--        </dependency>-->


        <!-- https://mvnrepository.com/artifact/junit/junit -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.13.2</version>
            <scope>test</scope>
        </dependency>

<!--        io.grpc.ManagedChannelProvider$ProviderNotFoundException: No functional channel service provider found. Try adding a dependency on the grpc-okhttp, grpc-netty, or grpc-netty-shaded artifact-->
<!--        at io.grpc.ManagedChannelProvider.provider(ManagedChannelProvider.java:45)-->


        <!-- https://mvnrepository.com/artifact/io.grpc/grpc-okhttp -->
        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-okhttp</artifactId>
            <version>1.49.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/io.grpc/grpc-netty -->
        <dependency>
            <groupId>io.grpc</groupId>
            <artifactId>grpc-netty</artifactId>
            <version>1.49.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.bouncycastle/bctls-fips -->
        <dependency>
            <groupId>org.bouncycastle</groupId>
            <artifactId>bctls-fips</artifactId>
            <version>1.0.13</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/io.netty/netty-tcnative-boringssl-static -->
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-tcnative-boringssl-static</artifactId>
            <version>2.0.25.Final</version>
        </dependency>


    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <!-- Run shade goal on package phase -->
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <artifactSet>
                                <excludes>
                                    <exclude>io.grpc:grpc-netty-shaded</exclude>

                                </excludes>
                            </artifactSet>
                            <!--                                <minimizeJar>true</minimizeJar>-->
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                        <exclude>META-INF/*.txt</exclude>
                                    </excludes>
                                </filter>
                                <!--                                    grpc-core-1.44.0.jar, grpc-grpclb-1.44.0.jar, grpc-services-1.44.0.jar, grpc-xds-1.44.0.jar define 1 overlapping resource-->
                                <filter>
                                    <artifact>io.grpc:grpc-grpclb</artifact>
<!--                                    grpc-grpclb-1.48.0.jar-->
                                    <excludes>
                                        <exclude>META-INF/services/io.grpc.LoadBalancerProvider</exclude>
                                        <exclude>META-INF/services/io.grpc.NameResolverProvider</exclude>
                                    </excludes>
                                </filter>

                                <filter>
<!--                                    grpc-services-1.48.0.jar-->
                                    <artifact>io.grpc:grpc-services</artifact>
                                    <excludes>
                                        <exclude>META-INF/services/io.grpc.LoadBalancerProvider</exclude>
                                        <exclude>META-INF/services/io.grpc.NameResolverProvider</exclude>
                                    </excludes>
                                </filter>

                                <filter>
<!--                                    grpc-xds-1.48.0.jar-->
                                    <artifact>io.grpc:grpc-xds</artifact>
                                    <excludes>
                                        <exclude>META-INF/services/io.grpc.LoadBalancerProvider</exclude>
                                        <exclude>META-INF/services/io.grpc.NameResolverProvider</exclude>
                                    </excludes>
                                </filter>

                                <filter>
<!--                                    grpc-googleapis-1.48.0.jar-->
                                    <artifact>io.grpc:grpc-googleapis</artifact>
                                    <excludes>
                                        <exclude>META-INF/services/io.grpc.NameResolverProvider</exclude>
                                    </excludes>
                                </filter>


                            </filters>
                            <transformers>
                                <!-- add Main-Class to manifest file -->
                                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                    <mainClass>com.economical.App</mainClass>
                                </transformer>
                            </transformers>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Weird this is it's working without any error from my local machine, the only different is it's using different version of java. on this java version it's working fine.

openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-39)
OpenJDK 64-Bit Server VM (build 17.0.1+12-39, mixed mode, sharing)

Upgrade to the latest Java 8. The one you have is rather old. — Thorbjørn Ravn Andersen, Sep 12 '22 at 19:52
@ThorbjørnRavnAndersen unfortunately that's not possible. I have to run this on the server where upgrading JAVA/JRE can cause hell lot of issues. — Gaurang Shah, Sep 13 '22 at 13:09
Not the latest version of Java but an updated and bugfixed Java 8 which is what is already there. 1.8_111 is _SIX_ years old - https://www.oracle.com/java/technologies/javase/8u111-relnotes.html - you may want to tell management that the deployment platform has not received updates for that long. They may want to prioritize it. — Thorbjørn Ravn Andersen, Sep 13 '22 at 13:49
@ThorbjørnRavnAndersen so what you are saying is updating it to `1.8.0_111-b14` might fix the issue. that might be doable. as this is hadoop environment, switching major version might have consequences — Gaurang Shah, Sep 13 '22 at 13:53
No. LATEST version. You should REALLY talk to those responsible for the hadoop environment... — Thorbjørn Ravn Andersen, Sep 13 '22 at 15:18
@ThorbjørnRavnAndersen thanks. However, that's not an option. this is hadoop we are using. changing the JRE/JDK would break more things then we can comprehend. — Gaurang Shah, Sep 13 '22 at 19:01

score 2 · Answer 1 · answered Sep 13 '22 at 18:30

2

As @Thorbjørn Ravn Andersen said, upgrade to the latest Java 8.

answered Sep 13 '22 at 18:30

Andrés

487
1
12

score 0 · Accepted Answer · answered Sep 14 '22 at 14:03

0

After doing a lot of search on google I landed on following page. https://webtide.com/jetty-alpn-java-8u252/

This explain the the support for ALPN was added in Java 8 252 version. However, there a workaround for older java version as well. exactly what I was looking for.

by adding following two arguments to my command now it's working without any error.

java -Xbootclasspath/p:/home/gaurnag/alpn-boot-8.1.13.v20181017.jar -javaagent:/home/gaurang/jetty-alpn-agent-2.0.10.jar -jar HDFSReplicator-1.0.jar

answered Sep 14 '22 at 14:03

Gaurang Shah

11,764
9
74
137

1

You should probably document this very, very thoroughly for the future maintainer of this code. Also the Hadoop team should strongly consider to start updating Java now instead of waiting until security has been breached. – Thorbjørn Ravn Andersen Sep 17 '22 at 13:59

Error Connecting to Secret Manager - TLS ALPN negotiation failed with protocols: [h2]

2 Answers2

Linked