We have a spark structured streaming application running in production with inhouse managed Kafka (lets call it kafka-inhouse
)
We are deciding to migrate to aiven kafka cloud
.
Assuming:
We consume all messages from
kafka-inhouse
, and then the new messages get published toaiven kafka cloud
The
path
and thecheckpoint
in S3 where we store the Kafka messages (using the spark streaming app), doesn't change.
Question:
Will the Spark structured streaming seamlessly manage change in Kafka brokers, especially the checkpointing
once we migrate?