0

I have looked into this article and still i have some confusion regarding merging separate topics in one comprehensive topic: https://www.confluent.io/blog/put-several-event-types-kafka-topic

So i have two entities, Account & Client as given below:

Account:

"Account": {
        "AccountId": "JKB123456",
        "ClientId": "1234567",
        "Type": "Savings",
        "Currency": "USD"
        }

Client:

"Client": {
        "ClientId": "1234567",
        "Name": "John Doe",
        "PhoneNo": "777777777"
        }

A Client can have at least one or more accounts associated with it. The events which will work on them ar pretty basic i.e. Create and Update.

This results in four use-cases:

  1. Create a client - A client entity with multiple Account entities are provided. If the client with all its account is created, the use-case is considered successfully executed.
  2. Update Client - A client (only) has some of its details updated, so a client object with the said values is provided. Account is not part of this.
  3. Create Account - An account entity is provided to be created and associated with an existing account.
  4. Update Account - An existing account fields can be updated in this use-case.

A create client json structure will look like this:

{
    "Client": {
        "ClientId": "1234567",
        "Name": "John Doe",
        "PhoneNo": "777777777",
        "Accounts": [
            {
                "AccountId": "JKB123456",
                "ClientId": "1234567",
                "Type": "Savings",
                "Currency": "USD"
            },
            {
                "AccountId": "HKB123456",
                "ClientId": "1234567",
                "Type": "Savings",
                "Currency": "EUR"
            }
        ]
    }
}

So should i have a single Client topic schema with list of account structure as part of it? I believe then i can publish both create and update events for the client on the same topic.

But would it be possible to publish the Account Create/update events on the same topic as well? Because i think if i do proper ordering (probably on the basis of clientid) then i can also have the all events of the same client on the same partition.

Do you think this is a good approach?

aran
  • 10,978
  • 5
  • 39
  • 69
Darsin
  • 123
  • 2
  • 11
  • You should keep in mind what unifying those messages would involve. For example, if you have some few clients with a lot more updates than others, and those are clients are assigned the same partition, would your partitions look evenly distributed? Sharing a topic also means sharing the retention policy; If this is based on the size, keep in mind you won't be able to store the messages for as long as you did before; If the retention is based on time, beware with the stored data size and your drives, as your topic will duplicate its size by the time retention kicks. – aran Mar 01 '21 at 18:17
  • Consider what involves this unification, both in the client side and broker side, and make your decission. Do you need both to be retained for the same time? Do you need the same replication level for both types? Is processing affected by this? And so on... You can always test if before putting in production. – aran Mar 01 '21 at 18:19
  • My suggestion if using this approach is creating two independent consumer groups; Each one will consume every message, but only the messages keyed as "update" get processed by one of them, and so on. Beware here with synchronization issues as well, could your consumers receive and update of an account, whose creation has not been received yet? – aran Mar 01 '21 at 18:21
  • thanks. i think i will start with single topic and multiple events. that way i retain the ordering of events. Probably evolve from there. – Darsin Mar 02 '21 at 12:05

0 Answers0