I am developing a consumer which consumes events from multiple Kinesis streams. I have some questions to understand the best practices.
Should I create one channel per stream? What factors should be considered to decide between "channel per stream" or "one channel for all streams"?
Which channel fits better for my case performance wise? There are different channel types like PollableChannel, SubscribaleChannel and DirectChannel.
Thank you
The KinesisMessageDrivenChannelAdapter is an active component and it performs consumption and message sending in the task executor. Therefore you might think do not shift messages to the QueueChannel or an ExecutorChannel - the logic is already async and involves enough threads on the machine. It is really much better do not shift the processing to a separate thread and keep this consumption thread busy and don't poll more records from the Kinesis into the memory.
One KinesisMessageDrivenChannelAdapter can do, essentially, the same work for several streams as several separate adapters for different stream - the thread capacity on the machine is going to be used.
We need different channel adapters in case of different processing logic or different data types, or different Kinesis Client options. In all other cases the single instance is pretty sufficient.
