I am reading data from Kaka topic and then doing a group by on the message received from topic val lines = spark.readStream.format("Kafka")... val df1 = lines.select($"timestamp", $"value".cast("STRING")) ...// Created a schema and fetched message from value and now df3 has schema timestamp and message string
Now, doing a group by on message to find the count and tried writing the result in file in append output mode. But it gave watermark error so in group by included timestamp like screenshot
My code snapshot
Can someone help how can I dump the count in file in append output mode
I expect the message count to be written in file for new data received in Kafka topic. Console mode is not suitable for my usecase
