7

We have a Data Processing Pipeline where we receive data from different sources. The entire pipeline is implemented by using Event Driven Architecture and microservices. One of the services has three separate tasks. Two of them are completely common across different sources of data, but the third task scope may change slightly depending on what our data source is. For example, for one source the unique signature may get calculated based on filed1 and filed2, and for another source, it may get calculated based on field2 and field3. How's the best way of implementing this service to be aligned with microservice granularity principles?

Three approaches have come to my mind:

1) Create a single service and use different implementation per source (like a factory design pattern). Depending on the source, one of the implementations will be used at the run-time.

Pros: Less number of services. Less complexity

Cons: Since the service will be shared across all data sources, by adding any new data source this service should be re-deployed which creates an implicit dependency between this service and any service responsible to collect data from a source

2) Break this service into two services, use one for all sources and reimplement the extracted service per data source.

Pros: No dependency between the collector and these two services. By adding a new source a new service needs to be implemented and it does not require to redeploy the services related to other sources.

Cons: More services and since the services are going to be too small we may end up facing nanoservice issue in future.

3) Do not change the granularity of services, but create different instances of the service at run-time. Each with a different config to specify what are the set of fields to be used for each source. In this case, the code is shared, but run-time instances are different depending on which source it belongs to.

Pros: Less number of service, and no operational dependency

Cons: Move complexity of logic to run-time which may make the deployment and operations more challenging

Ali
  • 1,759
  • 2
  • 32
  • 69
  • 1
    Have you taken into account scalability, criticality of each service, how frequently the new service would have to change, etc? Normally these NFR help defining whether you need to create a service or reuse an existing one (config/feature flags) – Amin Abu-Taleb Mar 13 '19 at 10:47
  • @AminAbu-Taleb Yes. Technically, the frequency of change is the main differentiation point here, as by adding a new source we need to modify this service which is impacted by solution 1. – Ali Mar 13 '19 at 11:09
  • is there a possibility to build a dynamic rules handler? I met similar problem in building a coupon system. BTW, i think the second one is unreasonable in an enterprise architecture. – wl.GIG Mar 29 '19 at 10:00
  • @wl.GIG What do you mean by a dynamic rule handler? – Ali Mar 29 '19 at 10:16
  • @Alin like a common handler for all data source with input parameters like [columns: {as, bb,...}, crypto: xxxxx ] – wl.GIG Mar 29 '19 at 10:21
  • @wl.GIG Isn't that solution 3? – Ali Mar 29 '19 at 10:40
  • @Alin it is kind of implement for s3 – wl.GIG Mar 29 '19 at 10:49
  • @wl.GIG My question is not really how to implement it. It is which solution at the high level and why. – Ali Mar 29 '19 at 10:51
  • How many different configurations do you expect? Is it possible that you get overwhelmed at some point with solution 3? – Nestor Sokil Mar 30 '19 at 13:07
  • @NestorSokil I cannot predict. Right now, 7-8 different versions, but in future, it may become 50 – Ali Mar 31 '19 at 08:59
  • @Alin and are these processing jobs consuming events from different topics or single topic+different groups? What broker do you use? – Nestor Sokil Mar 31 '19 at 09:36
  • @NestorSokil It is up to us how to design it. It can be a single topic or multiple topics. We use Kafka. – Ali Mar 31 '19 at 11:44

3 Answers3

0

It depends. FYI, I know of Kafka but don't have experience in it.

For option 3: how mature is your ability to manage various instances of the solution with different configuration? How many instances would there be? How would you be able to observe the health, behavior and performance of all the instances?

This option means the development is less complex but the operational side is more complex: you only have one code-base to test but lots of different config permutations to test.

For option 1: would be more complex on the development side, and would still have some complexity for operations. The reason for that would depend on how the solution is set to recognize which implementation to use at runtime. An IOC approach would work but it's still config that needs to be managed.

For both approaches you'd be able to set-up an instance with the right configuration and test it, which is good.

Regarding System Change: microservices should ideally be easily replaceable. Make sure the delineation between services is clean, so that you can ideally replace one part of the solution without breaking the other. It should also be easy to test - both testing the service/module in isolation, and an integrated test with the rest of the solution - and with the relevant configuration - before deploying such a change into production. If you feel one approach is better suited to this that the other then that's the approach I would probably go with.

Update - Option 2

That means having multiple services to handle some requests (but not others) so now you have to manage both the flow between services (at runtime), and also the interdependencies between them (at development, deployment time); harder to test.

Microservices is partly about having services which are independent - option two is not really in-line that ethos - which is ok, just realize it's not strictly a microservice, depending on how particular you are about such things.

Adrian K
  • 9,880
  • 3
  • 33
  • 59
0

I agree with Adrian, it depends on your situation.

I think the most important is system complexity - it plays a critical role in testing, support, and evolution of the system. (KISS)

So I think the best option is 2.

Of course, you should remember other design principles - create libraries for reusing and so on, but in any case, your source code design will be tree not net with unmanaged dependencies.

A Ralkov
  • 1,046
  • 10
  • 19
0

I think I would bother most about the type of result the service(s) produce. If (for all 3 options) the result of the process would be the very same type (or domain object), I surely would favor Option 1.

This would align with the "loose coupling, cohesive behavior" principle, Sam Newman wrote about in his book.

Loose coupling because consumers don't have to care about what service to call/receive events from and therefore being relieved of knowing the source.

Cohesive behavior because a single service applies business logic to events, and therefore produces reliably cohesive events with consistent type/semantics.

For the different sources I would say that it is okay to let them be consumed by a single service. There are quite some possibilities to design it in a maintainable way inside in the service implementation. What task should be chosen by source is an implementation detail that should be hidden from consumers.

Cons: Since the service will be shared across all data sources, by adding any new data source this service should be re-deployed which creates an implicit dependency between this service and any service responsible to collect data from a source

This Service would indeed depend on a producer because it needs it as a source. There is no way to circumvent that. The same would be true for all other options to some extend. But the dependency is consumer depends on producer, not the other way around!

sschrass
  • 7,014
  • 6
  • 43
  • 62