0

I am using PCF to run my spring-boot application. Also used the PCF RabbitMQ service.

My problem is there is some time that the message in queue was pinned long time in unacked and I noticed that the number of running app instances in PCF is less than the number of active consumer channel in RabbitMQ.

RabbitMQ

PCF

I also have autoscaler enabled that will automatically scale up/down the app instance depending on the applied rule.

I have no luck replicating the issue as there are no error logs in the client app and the only solution that I did is to restart the application in PCF. I also tried to force kill the application and it seems that the consumer channel is destroyed as well.

Any help or suggestion that I can look up? I used spring cloud stream to my consumer app and very basic config in yml file (no dlq, no custom timeout, etc)

Thanks in advance!

jitd
  • 45
  • 1
  • 10

2 Answers2

1

2.0.x is no longer supported; however, the latest 2.0.x is 2.0.14.

If you are using auto recovery in the amqp-client (not recommended because Spring AMQP has its own recovery mechanisms pre-dating autorecovery in the client), there was a fix included in 2.0.7.

You should at least upgrade to 2.0.14, but preferably to a newer supported release; the current release is 2.2.10.

That said, the vast majority of issues like this are caused by the listener thread being "stuck" in user code; take a thread dump the next time it happens.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • Thank you for the reply, We will monitor our app and get the thread dump next time. Hope to find something. Will also try to upgrade the spring-amqp version we are using. – jitd Aug 20 '20 at 08:00
1

We had similar issue, as far as I remember, we had to set automaticRecoveryEnabled to false on rabbit connection factory as there was a conflict between this one and spring autorecovery.

Dim D
  • 61
  • 6
  • The rabbit connection factory in the version that we are using is setting the automaticRecoveryEnabled to false by default. – jitd Aug 24 '20 at 08:26
  • Found something, there is an artifact (spring_auto_reconfiguration-2.XX.X_RELEASE.jar) running in PCF along with spring boot application that auto configure connection factory and make the automaticRecoveryEnabled value true. – jitd Aug 25 '20 at 15:18