2

I have an Amazon Kinesis stream, consisting of multiple shards. The number of shards, and therefore the number of consumers, is not a constant.

There is an infrequent type of event that I want broadcasted to every consumer on the stream.

Is there a way for a producer to broadcast a record, i.e. to discover the shards and put the record on each one?

njasi
  • 126
  • 8
Chris
  • 419
  • 5
  • 13
  • The word "consumer" in your question is vague. Do you mean "every consumer application" or "every worker thread listening to a shard in one application"? – az3 Jan 20 '16 at 13:32
  • I have one application and multiple instances of that app. say 4 and I want same message to be consumed by 4 instances, how can I achieve this in kinesis data stream – Rahul Singh May 12 '20 at 17:10

1 Answers1

1

You can do this! Kind of...

The trick it to use the parameter "ExplicitHashKey".

This lets you set the hash key used for the record, and therefore lets you chose what shard your data is going on.

You can then call

aws kinesis describe-stream --stream-name name-of-your-stream

to get info about each shard including what hash range each shard covers.

Then you only need to send your data to each shard (with PutRecord or PutRecords) and you are all set.

Unfortunately, there is no "broadcast" but you can easily write a Lambda to send the same data to each shard using the ExplicitHashKey parameter.

Another solution is to use separate streams and write out to each one.

Mike
  • 1,727
  • 2
  • 19
  • 32