1
22/11/09 11:08:40 INFO MicroBatchExecution: Resuming at batch 206 with committed offsets {KafkaV2[Subscribe[test]]: 
{"test":{"0":3086,"1":3086,"2":3086,"3":3086,"4":3086,"5":3086,"6":3086,"7":3086,"8":3086,"9":3086,"10":3086,
"11":3086,"12":3086,"13":3086,"14":3086,"15":3086,"16":3086,"17":3086,"18":3086,"19":3086,"20":3086,"21":3086,"22":3086,"23":3086,
24":3086,"25":3086,"26":3086,"27":3086,"28":3086,"29":3086,"30":3086,"31":3886,"32":3086,"33":3086,"34":3086,"35":3086,"36":3086,
"37":3086,"38":3086,"39":3086,"40":3086,"41":3086,"42":3086,"43":3086,"44":3086,"45":3086,"46":3086,"47":3086,"48":3086,"49":3086}}} 
and available offsets {KafkaV2[Subscribe[test]]: {"test":{"0":3105,"1":3105,"2":3105,"3":3105,"4":3105,
"5":3105,"6":3105,"7":3105,"8":3105,"9":3105,"10":3105,"11":3105,"12":3105,"13":3105,"14":3105,"15":3105,"16":3105,"17":3105,"18":3105,"19":3105,
"20":3105,"21":3105,"22":3105,"23":3105,"24":3105,"25":3105,"26":3105,"27":3105,"28":3105,"29":3105,"30":3105,"31":3910,"32":3105,"33":3105,"34":3105,
"35":3105,"36":3105,"37":3105,"38":3105,"39":3105,"40":3105,"41":3105,"42":3105,"43":3105,"44":3105,"45":3105,"46":3105,"47":3105,"48":3105,"49":3105}}}

This offset collection is spark streaming checkpoint offsets. How can i get this offset collection by code? Is any property in spark streaming?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
qtcat
  • 15
  • 5
  • Docs suggest using AdminClient https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#offset-fetching – OneCricketeer Nov 10 '22 at 14:13

0 Answers0