8

I would like to know/get opinions on how to setup liveness probe for a RabbitMQ queue consumer. I am not sure how to verify if consumer is still processing messages from the queue. I have already tried searching for some clues over the internet but could not find any. So just asking a question here to see if anyone has got any idea.

The code block which I want to make sure working fine is

                var consumer = new EventingBasicConsumer(channel);
                consumer.Received += (model, ea) =>
                {
                    var body = ea.Body;
                    var message = Encoding.UTF8.GetString(body);
                    Console.WriteLine($"Message Received: {message}");
                };

Thank you.

Alpesh
  • 606
  • 6
  • 15
  • Are you trying to apply liveness probe in the yaml/json configuration file for RabbitMQ? – Shudipta Sharma Nov 19 '18 at 14:11
  • Also show your docker image used for RabbitMQ. Have you used the official docker image `rabbitmq`? – Shudipta Sharma Nov 19 '18 at 14:17
  • 1
    @ShudiptaSharma I have got rabbitmq container covered with both readiness and liveness probes. This question is about message consumer which is hosted in a different container. – Alpesh Nov 19 '18 at 17:35

1 Answers1

3

First thing, you will need to expose an HTTP endpoint in your application code that checks whether the consumer is alive or not.

There are many ways to test the liveness of the consumer, for example, you can check the timestamp of the last message that was consumed. If it's too old, you could declare the consumer as dead by returning an HTTP 500 error, otherwise, return HTTP 200. It depends on your business logic, you might want to use what I proposed, or any other method that fits your needs.

Once you have an HTTP endpoint, you can define a liveness probe in your Kubernetes manifest.

livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
        httpHeaders:
        - name: X-Custom-Header
          value: Awesome
      initialDelaySeconds: 3
      periodSeconds: 3

(Taken from https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/)

areller
  • 4,800
  • 9
  • 29
  • 57
  • thanks for your answer. I have used all three types of probes in other parts of my AKS. I can setup probe but not sure what the logic should be to check if consumer is still processing messages. I liked your idea of checking timestamp but I am looking for a solution which can more accurately confirm the liveness of message consumer. It looks like I have to go for some sort of indirect solution similar to one you have suggested. – Alpesh Nov 19 '18 at 17:32
  • @Alpesh depending on what language you're using, the RabbitMQ client might have a method that returns whether it's alive or not. A quick search of the Java client led me to this document: https://rabbitmq.github.io/rabbitmq-java-client/api/current/com/rabbitmq/client/Consumer.html You can subscribe to events that indicate the state of the consumer. If you want to be sure, you could combine the solution I suggested with listening to these events. – areller Nov 19 '18 at 17:39
  • Not sure if any of the methods on the page can help in my scenario. I am using C# client and can't find anything useful there as well. I will keep looking for something useful, – Alpesh Nov 19 '18 at 18:02
  • You have events that correspond to these methods https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/master/projects/client/RabbitMQ.Client/src/client/events/EventingBasicConsumer.cs. Note that there is a default heartbeat mechanism in the consumer https://stackoverflow.com/questions/33699165/rabbit-mq-recovery-of-connection-channel-consumer So I don't think that you would gain anything by handling these events. You could still use the method that I've proposed just to make sure. – areller Nov 19 '18 at 18:36
  • There are three types to define a probe(readiness, lives, startup), they are HTTP, Command, TCP. The one which suits workers is the command one. – DaAmidza Oct 20 '20 at 11:39
  • @DaAmidza not necessarily.. sure, if you have a way to probe your service from the command line, otherwise, HTTP would be simpler for most .NET services – areller Oct 20 '20 at 17:30
  • 1
    @areller it's not rocket science to create a sh script which will test it for the case. RabbitMQ suggests testing it through the command line. Services that are exposed to the web, yes they can go over the HTTP way. But this case MUST go with the command. Makes no sense to explose it to HTTP if you don't need it. – DaAmidza Oct 22 '20 at 07:29
  • @DaAmidza how is your sh script going to communicate with the service though? In many cases you'd probably want to get metrics beyond whether or not the process is running. Also, if you're using ASP.NET Core to host your RabbitMQ consumer, adding an HTTP healthcheck point is pretty straightforward https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks?view=aspnetcore-3.1 – areller Oct 22 '20 at 14:00