1

We have a spark structured streaming application running in production with inhouse managed Kafka (lets call it kafka-inhouse)

We are deciding to migrate to aiven kafka cloud.

Assuming:

  1. We consume all messages from kafka-inhouse, and then the new messages get published to aiven kafka cloud

  2. The path and the checkpoint in S3 where we store the Kafka messages (using the spark streaming app), doesn't change.

Question:

Will the Spark structured streaming seamlessly manage change in Kafka brokers, especially the checkpointing once we migrate?

zero323
  • 322,348
  • 103
  • 959
  • 935
Chandan Bhattad
  • 351
  • 1
  • 5
  • 21

0 Answers0