5

I have an Azure function with ServiceBusTrigger which will post the message content to a webservice behind an Azure API Manager. In some cases the load of the (3rd party) webserver backend is too high and it collapses returning error 500.

I'm looking for a proper way to implement circuit breaker here.

I've considered the following:

  • Disable the azure function, but it might result in data loss due to multiple messages in memory (serviceBus.prefetchCount)
  • Implement API Manager with rate-limit policy, but this seems counter productive as it runs fine in most cases
  • Re-architecting the 3rd party webservice is out of scope :)
  • Set the queue to ReceiveDisabled, this is the preferred solution, but it results in my InputBinding throwing a huge amount of MessagingEntityDisabledExceptions which I'm (so far) unable to catch and handle myself. I've checked the docs for host.json, ServiceBusTrigger and the Run parameters but was unable to find a useful setting there.
  • Keep some sort of responsecode resultset and increase retry time, not ideal in a serverless scenario with multiple parallel functions.
  • Let API manager map 500 errors to 429 and reschedule those later, will probably work but since we send a lot of messages it will hammer the service for some time. In addition it's hard to distinguish between a temporary 500 error or a consecutive one.

Note that this question is not about deciding whether or not to trigger the circuitbreaker, merely to handle the appropriate action afterwards.

Additional info

  • Azure functionsV2, dotnet core 3.1 run in consumption plan
  • API Manager runs Basic SKU
  • Service Bus runs in premium tier Messagecount: 300.000
Edwin
  • 51
  • 2
  • I have to solve this same issue. I too like the idea of ReceiveDisabled, perhaps in conjunction with shutting the function down after some quiesce of the in-memory processing. This may need to be a coordinated action. One question is: who would initiate the circuit breaker, only the function knows when to trigger it. – JohnKoz Jun 04 '20 at 21:05
  • I solved it by injecting a singleton custom Circuitbreaker service into my function. All function instance can check the circuitbreaker state and decide their actions. It ain't perfect though as the circuitbreaker instance works only within triggers on the same host. So lets say you have 2 hosts which all execute 10 requests you end up with 3 different circuitbreaker instances. Since they all need to reach their "open" state it will fire a couple of additional requests to your backend – Edwin Jun 06 '20 at 10:01
  • Hey @Edwin did you take this any further? I'm looking to implement something similar by disabling queue delivery. – Nick May 14 '21 at 10:21
  • 1
    @Nick: disabling queue delivery might be a bit easier. Although the subscriber function requires an actieve queue, you can create a topic with a forward queue. It will result in the following: topic -> topicsubscriber -> forward queue -> queue subscriber You can then disable the specific topicsubscriber or the whole topic in general. Note that if you disable a subscription, the message will get lost if you don't implement a way to catch it (2nd subscriber or DLQ) – Edwin May 16 '21 at 06:41
  • @Edwin: Yeah, it's the fact that a `ServiceBusTrigger` requires an `Active` queue which is the pain-point. If the servicebus listener behind the scenes dealt with the situation more gracefully then this would definitely be the way to implement a circuit-breaker. I modified the `ServiceBusTrigger` to receive a batch of messages and this seemed to handle the `Disabled` state much more gracefully but I'm wary of relying on that given the issues with the single-message trigger. (I only tested it locally - not on Azure) – Nick May 18 '21 at 14:07
  • You can slow down processing by Task.Delay and let the message lock renew itself, until it reach max renew limit whatever you have set it to. I would not suggest this in a consumption based scenario where you pay pr. millisecond execution time. By stopping the function no data should be lost as long as a message has not been completed it will remain on the service bus. – CodeMonkey Dec 13 '21 at 07:42

0 Answers0