0

I have Spark job reads millions of records from Cassandra, filter out(business rules) and write to Kinesis stream. I don't find any example and testimonial on how to invoke KPL(Kinesis Producer Library) from Spark. Is that correct approach? Do I have any other option?

ameenun
  • 1
  • 1

1 Answers1

0

you can create KPL producer per partition and then for each partition you can send the message. Keep the partitions small to avoid overloading task/core nodes.

Aniket
  • 1