I'm trying to communicate via grpc between two microservices internally on kubernetes, but I'm getting a connection refused error.
These are the yaml files of the services that are trying to communicate.
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: source-api
spec:
ports:
- name: grpc-server
port: 8081
protocol: TCP
targetPort: 8081
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: source-api
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
template:
metadata:
labels:
app.kubernetes.io/name: source-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
spec:
containers:
- env:
- name: QUARKUS_GRPC_CLIENTS_STORE_PORT
value: "8081"
- name: QUARKUS_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
key: datasourcePassword
name: liop
- name: KAFKA_BOOTSTRAP_SERVERS
value: kafka-service:9092
- name: QUARKUS_DATASOURCE_USERNAME
valueFrom:
secretKeyRef:
key: datasourceUsername
name: liop
- name: QUARKUS_HTTP_PORT
value: "8080"
- name: QUARKUS_GRPC_SERVER_PORT
value: "8081"
- name: QUARKUS_GRPC_SERVER_HOST
value: localhost
- name: QUARKUS_DATASOURCE_JDBC_URL
value: jdbc:mysql://mysql:3306/product
- name: QUARKUS_GRPC_CLIENTS_STORE_HOST
value: store-api
image: tools_source-api:latest
imagePullPolicy: Never
name: source-api
ports:
- containerPort: 8081
name: grpc-server
protocol: TCP
- containerPort: 8080
name: http
protocol: TCP
imagePullSecrets:
- name: gitlab-registry
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: product-api
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: product-api
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
template:
metadata:
labels:
app.kubernetes.io/name: product-api
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
spec:
containers:
- env:
- name: KAFKA_BOOTSTRAP_SERVERS
value: kafka-service:9092
- name: QUARKUS_DATASOURCE_JDBC_URL
value: jdbc:mysql://mysql:3306/product
- name: QUARKUS_GRPC_CLIENTS_IMAGE_PORT
value: "8081"
- name: QUARKUS_GRPC_CLIENTS_SOURCE_HOST
value: source-api
- name: QUARKUS_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
key: datasourcePassword
name: liop
- name: QUARKUS_DATASOURCE_USERNAME
valueFrom:
secretKeyRef:
key: datasourceUsername
name: liop
- name: QUARKUS_GRPC_CLIENTS_SOURCE_PORT
value: "8081"
- name: QUARKUS_GRPC_CLIENTS_IMAGE_HOST
value: media-api
image: tools_product-api:latest
imagePullPolicy: Always
name: product-api
ports:
- containerPort: 8080
name: http
protocol: TCP
imagePullSecrets:
- name: gitlab-registry
This is the yaml of my API-gateway, which does correctly communicate via HTTP with the microservices:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/version: latest
app.kubernetes.io/part-of: liop
app.kubernetes.io/name: api-gateway
name: api-gateway
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/version: latest
app.kubernetes.io/part-of: liop
app.kubernetes.io/name: api-gateway
template:
metadata:
labels:
app.kubernetes.io/version: latest
app.kubernetes.io/part-of: liop
app.kubernetes.io/name: api-gateway
spec:
containers:
- env:
- name: product_api
value: http://product-api:8080/api/products/v1/
- name: source_api
value: http://source-api:8080/api/sources/v1/
- name: store_api
value: http://store-api:8080/api/stores/v1/
- name: report_api
value: http://report-api:8080/api/reports/v1/
- name: category_api
value: http://category-api:8080/api/categories/v1/
- name: AUTH0_ISSUER_URL
value: xxxx
- name: AUTH0_AUDIENCE
value: xxxxxxx
- name: PORT
value: "7000"
image: tools_webgateway:latest
imagePullPolicy: Never
name: api-gateway
ports:
- containerPort: 7000
hostPort: 7000
name: http
protocol: TCP
imagePullSecrets:
- name: gitlab-registry
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/name: api-gateway
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
name: api-gateway
spec:
ports:
- name: http
port: 7000
protocol: TCP
targetPort: 7000
selector:
app.kubernetes.io/name: api-gateway
app.kubernetes.io/part-of: liop
app.kubernetes.io/version: latest
Error the product-api throws:
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:533)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.quarkus.grpc.runtime.supports.IOThreadClientInterceptor$1$1.onClose(IOThreadClientInterceptor.java:68)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: source-api/10.101.237.82:8081
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
The source-api:
source-api ClusterIP 10.101.237.82 <none> 8081/TCP,8080/TCP
2021-04-01 06:38:08,973 INFO [io.qua.grp.run.GrpcServerRecorder] (vert.x-eventloop-thread-1) gRPC Server started on localhost:8081 [SSL enabled: false]
Using grpcurl internally also gives me a connection refused error. But port forwarding the source-api:8081 does allow me to do requests.
