8

We have the first version of an application based on a microservice architecture. We used REST for external and internal communication.

Now we want to switch to AP from CP (CAP theorem)* and use a message bus for communication between microservices. There is a lot of information about how to create an event bus based on Kafka, RabbitMQ, etc. But I can't find any best practices for a combination of REST and messaging. For example, you create a car service and you need to add different car components. It would make more sense, for this purpose, to use REST with POST requests. On the other hand, a service for booking a car would be a good task for an event-based approach.

Do you have a similar approach when you have a different dictionary and business logic capabilities? How do you combine them? Just support both approaches separately? Or unify them in one approach?

* for the first version, we agreed to choose consistency and partition tolerance. But now availability becomes more important for us.

www.admiraalit.nl
  • 5,768
  • 1
  • 17
  • 32
  • I find this question hard to understand. Do you mean "AP from CP" instead of "AP form CP"? Could you please improve this sentence: " I can't see any reason to do it with an event (..) but create a booking for car service good task for event-based approach." What do you mean by "How you combine them"? Do you want to implement both a REST-API and a non-REST-API? – www.admiraalit.nl Jun 11 '19 at 15:14
  • @www.admiraalit.nl thanks for the review. I've edited. – Artem Antonenko Jun 11 '19 at 18:34
  • 1
    If you think of CQRS (Command Query Responsibility Segregation) as a split between Event Driven (Command) and REST Based (Query), then you see how different operations can use different methods of communciating. I wouldn't mix both in a given operation, since REST is a blocking protocol and events are by their nature meant to be non-blocking and asynchronous. – George Stocker Jun 11 '19 at 18:50
  • 1
    Between microservices , you never use Post request. Like someone mentioned REST/HTTP is blocking protocol . So you can't scale well. Generally you would use SAGA pattern. You commit the transaction locally and emit the event on message broker. The event will be consumed by other services. If there is any problem you need to raise compensated transaction/Event. You also save the events locally before publishing them call outbox pattern. So you don't loose events if service bus is not available https://microservices.io/patterns/data/saga.html – Imran Arshad Jun 12 '19 at 02:56

1 Answers1

17

Bottom line up front: You're looking for Command Query Responsibility Segregation; which defines an architectural pattern for breaking up responsibilities from querying for data to asking for a process to be run. The short answer is you do not want to mix the two in either a query or a process in a blocking fashion. The rest of this answer will go into detail as to why, and the three different ways you can do what you're trying to do.

This answer is a short form of the experience I have with Microservices. My bona fides: I've created Microservices topologies from scratch (and nearly zero knowledge) and as they say hit every branch on the way down.

One of the benefits of starting from zero-knowledge is that the first topology I created used a mixture of intra-service synchronous and blocking (HTTP) communication (to retrieve data needed for an operation from the service that held it), and message queues + asynchronous events to run operations (for Commands).

I'll define both terms:

Commands: Telling a service to do something. For instance, "Run ETL Batch job". You expect there to be an output from this; but it is necessarily a process that you're not going to be able to reliably wait on. A command has side-effects. Something will change because of this action (If nothing happens and nothing changes, then you haven't done anything).

Query: Asking a service for data that it holds. This data may have been there because of a Command given, but asking for data should not have side effects. No Command operations should need to be run because of a Query received.

Anyway, back to the topology.

Level 1: Mixed HTTP and Events

For this first topology, we mixed Synchronous Queries with Asynchronous Events being emitted. This was... problematic.

Message Buses are by their nature observable. One setting in RabbitMQ, or an Event Source, and you can observe all events in the system. This has some good side-effects, in that when something happens in the process you can typically figure out what events led to that state (if you follow an event-driven paradigm + state machines).

HTTP Calls are not observable without inspecting network traffic or logging those requests (which itself has problems, so we're going to start with "not feasible" in normal operations). Therefore if you mix a message based process and HTTP calls, you're going to have holes where you can't tell what's going on. You'll have spots where due to a network error your HTTP call didn't return data, and your services didn't continue the process because of that. You'll also need to hook up Retry/Circuit Breaker patterns for your HTTP calls to ensure they at least try a few times, but then you have to differentiate between "Not up because it's down", and "Not up because it's momentarily busy".

In short, mixing the two methods for a Command Driven process is not very resilient.

Level 2: Events define RPC/Internal Request/Response for data; Queries are External

In step two of this maturity model, you separate out Commands and Queries. Commands should use an event driven system, and queries should happen through HTTP. If you need the results of a query for a Command, then you issue a message and use a Request/Response pattern over your message bus.

This has benefits and problems too.

Benefits-wise your entire Command is now observable, even as it hops through multiple services. You can also replay processes in the system by rerunning events, which can be useful in tracking down problems.

Problems-wise now some of your events look a lot like queries; and you're now recreating the beautiful HTTP and REST semantics available in HTTP for messages; and that's not terribly fun or useful. As an example, a 404 tells you there's no data in REST. For a message based event, you have to recreate those semantics (There's a good Youtube conference talk on the subject I can't find but a team tried to do just that with great pain).

However, your events are now asynchronous and non-blocking, and every service can be refactored to a state-machine that will respond to a given event. Some caveats are those events should contain all the data needed for the operation (which leads to messages growing over the course of a process).

Your queries can still use HTTP for external communication; but for internal command/processes, you'd use the message bus.

I don't recommend this approach either (though it's a step up from the first approach). I don't recommend it because of the impurity your events start to take on, and in a microservices system having contracts be the same throughout the system is important.

Level 3: Producers of Data emit data as events. Consumers Record data for their use.

The third step in the maturity model (and we were on our way to that paradigm when I departed from the project) is for services that produce data to issue events when that data is produced. That data is then jotted down by services listening for those events, and those services will use that (could be?) stale data to conduct their operations. External customers still use HTTP; but internally you emit events when new data is produced, and each service that cares about that data will store it to use when it needs to. This is the crux of Michael Bryzek's talk Designing Microservices Architecture the Right way. Michael Bryzek is the CTO of Flow.io, a white-label e-commerce company.

If you want a deeper answer along with other issues at play, I'll point you to my blog post on the subject.

George Stocker
  • 57,289
  • 29
  • 176
  • 237
  • Thank you for so detailed answer. During discussions, we decided to use something similar to the 3rd level. And thanks for links, it should help us. – Artem Antonenko Jun 12 '19 at 14:03
  • When you say that "External customers still use HTTP; but internally you emit events when new data is produced", I'm curious what the ramifications of this are on the architecture of an individual service. E.g. it sounds like many services will need to be listening for both HTTP requests (to support the rest endpoints) as well as be listening for events from some messaging service (e.g. redis). Does that mean you would need to run two processes for this service (e.g. a traditional REST api, and a separate process listening to the event bus), or is there a way to integrate both into one? – G. Shand Jul 13 '20 at 20:03
  • 2
    @G.Shand You can use the same service; and effectively have two 'endpoints'; a message queue listener; and an HTTP API endpoint; and both end up calling the same method internally to make something happen. The point here is to decouple what the service does from the delivery mechanism it does it for. – George Stocker Jul 13 '20 at 21:55