0

thinking about scaling, i wonder how queries are distributed among KSQL Servers ?

1 - Let say you have the cli or confluent control center, does the cli connect to all server or one of them ? Can it send the query to multiple KSQL SERVER e.g. those with the same id (i.e Pool id) ?

2 - Can we have multiple instance of the same query running on multiple KSQL SERVER ? if yes how does that work, e.g. with respect to consumer group or partitions ?

3 - Can we have multiple instance of the same query running on the same KSQL SERVER ?

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92
MaatDeamon
  • 9,532
  • 9
  • 60
  • 127
  • There's a lot of overlap with your subsequent question here : https://stackoverflow.com/questions/56926837/ksql-server-elastic-scaling-in-kubernetes Can you edit this one to reflect your remaining questions? – Robin Moffatt Jul 08 '19 at 08:41
  • After readying around, i guess there is a lot to unpack in my question, will further clarify it. – MaatDeamon Jul 09 '19 at 08:54

1 Answers1

0

@Robin Moffatt please correct if i am wrong:

1 - Let say you have the cli or confluent control center, does the cli connect to all server or one of them ? Can it send the query to multiple KSQL SERVER e.g. those with the same id (i.e Pool id) ?

It seems like not. One connect to one KSQL SERVER, and the command will be forwarded to the Command Topic, which will be red by all KSQL SERVER.

2 - Can we have multiple instance of the same query running on multiple KSQL SERVER ? if yes how does that work, e.g. with respect to consumer group or partitions ?

Yes, the query is send to one KSQL SERVER, which forward it to the command topic, from which other KSQL SERVERs picks up and run. I suspect there must be coordination, to avoid over-provisioning the query. Otherwise if each Server Automatically run the command, we may end with more topology instance than there is partition. Can you clarify that last part ?

3 - Can we have multiple instance of the same query running on the same KSQL SERVER ?

No we can't. Only one instance of a query per server, however, that query my be set to have a certain number of Thread. Each Thread is "some sort of independent instance of the application", or a unit of parallelism in Kafka Jargon. In fact i would say more specifically, each Thread is an independent instantiation of the Topology.

MaatDeamon
  • 9,532
  • 9
  • 60
  • 127