3

We are using Node.js, Docker Swarm, and GRPC to deploy multiple microservices. One of our microservices is a GRPC service that handles websockets for our frontend. However, when we make a GRPC call to this service from another microservice (Our backend service), we're getting an error that says "CANCELLED: Call cancelled" randomly. We've tried debugging this issue, but we're unable to find the root cause.

Here are some details about our setup:

  • We're using Docker Swarm to manage our microservices.

  • We have one replica of each service, we had multiple replicas earlier, but we reduced it in hope of solving this.

  • Our frontend communicates with the service using sockets.

  • We're making the GRPC call from another microservice that's also running on Docker Swarm.

We use GCP VM with 4 vCpu and 16gb ram, and deploy the docker swarm on this.

We're having trouble debugging this issue which happens randomly and would appreciate any help.

Here's what we've already tried:

  • We checked the logs of the GRPC service, but we couldn't find any relevant error messages.

  • We tried deploying multiple replicas of the GRPC service, but that didn't help either.

Nishantc7
  • 41
  • 2

1 Answers1

0

The "Call cancelled" error indicates that the client received a cancellation for that request. The Node.js gRPC server can send cancellations on active requests if the method forceShutdown is called. Otherwise, there is a good chance that the cancellation is coming from whatever load balancer you are using to distribute requests/connections across the replicas of your service.

murgatroid99
  • 19,007
  • 10
  • 60
  • 95