-1

I have 2 services running with spring boot and I have the following situation:

Service A (N instances) -> Queue (multiple instances) -> Service B (N instances)

  • Service A enqueue events async
  • Service B dequeue events

Traffic is increasing and we've noticed that events spend a lot of time in queue. We need to process events faster. Payload for each event is small and this solution has been working for some years now and a couple of years ago they thought that having a queue was a good idea but now I'm having this performance issue.

  • I thinking about creating an endpoint in service B and hit this endpoint from service A.

This call should be async and also implement a circuit breaker to avoid lossing messages if B goes down. If B goes down I could use the queue to keep messages and once B is running and up again pull messages from queue.

I have 2 questions:

  • Is it possible to implement circuit breaker or a failover mechanism for an async call?

  • do you think there is some other approach which could be better?

Thanks

Justin Bertram
  • 29,372
  • 4
  • 21
  • 43
Julio
  • 429
  • 2
  • 16
  • 1
    how your solution will help with `We need to process events faster.`? maybe you need to look into service b performance/scaling? – Iłya Bursov Jul 08 '21 at 18:49
  • We have some monitors in the services and queue and we see messages spend most of the time in queue. When we increase instances in queue everything works better – Julio Jul 08 '21 at 18:52
  • So that means service B process everything fast as no need to add more instances in B once we increase instances in the queue – Julio Jul 08 '21 at 18:53
  • you can use consumergroups for scaling and consuming messages.Link::https://activemq.apache.org/message-groups – Vaibs Jul 09 '21 at 05:26
  • Asking for "some other approach which could be better" is going to generate discussion and opinion-based answers rather than fact-based answers. This is off-topic here on Stack Overflow. Furthermore, you asking multiple questions here which is also off-topic. – Justin Bertram Jul 09 '21 at 16:56

3 Answers3

0

for question

  1. yes, it is possible, I am assuming you are using spring boot as you have mentioned java in tags, so there is a mechanism called RetryTemplate that you can use for async calls as well, you can find more details here -> https://www.openprogrammer.info/2019/09/03/using-spring-boot-async-with-retry/

  2. There are better approaches I would say, what did you implement when you say queue here? like is it a LIFO or something like AWS SQS? if that is the case then you can try and look for topic-based queue mechanisms like apache kafka

Hardik Uchdadiya
  • 242
  • 1
  • 8
  • We use active mq... then the quetion would be, do you think async calls would be faster than any other queue system or similar like sqs, kafka, etc – Julio Jul 08 '21 at 18:57
  • so it would not be faster than queues for sure, but at the same time queues provide you confirmed delivery of every message you put in, you don't need to rely on circuit breakers. And you can always fine tune the queue parameters for better performance – Hardik Uchdadiya Jul 08 '21 at 19:01
  • Sorry i didn't get it you are saying that queues are faster than call between services async? – Julio Jul 08 '21 at 19:11
  • oh, I typed "not" by mistake, it would be faster than queues for sure, but queues are reliable in terms of confirmed message deliveries as I mentioned earlier – Hardik Uchdadiya Jul 09 '21 at 06:24
0

If the bottleneck is in the queue, I don't think you should just remove it. You can just try to use a SQS or other cloud-based queue, which doesn't have that problem.

What happens if service A calls service B directly, and service B is down? The request will be lost. With a queue, the message will remain there until service B recovers.

Circuit breakers are used to avoid overwhelming a failed service with requests (in order to allow it to recover). So when it kicks in, there is a clear data loss there.

Cosmin Ioniță
  • 3,598
  • 4
  • 23
  • 48
  • Actually we are hit the endpoint directly and implement circuit braker... if B goes down we'll enqueue events in the queue. Also we'll replace active mq by sqs – Julio Jul 09 '21 at 18:45
0

Late to the party . But in this case I am not sure what you are trying to solve .

  • Are these requests time sensitive ?
  • Do these queues have hard limits that you are reaching
  • Is the queue size growing for all the instances?
  • How is the Service B consuming from the queue are there dedicated instances per queue or they all poll at random ?

Based on this

  • Scale up B
  • Shard queues to hosts to avoid noisy neighbor problem

Anti patterns

  • Asking producer to slow down is just passing your problem to them, They need to figure out a solution now. :)
  • Since this is Async unless you really are reaching queue limits or have reliability with queue instances you should be okay.
RandomGuy
  • 81
  • 4