I am trying to figure out how to write a Spark Stream to a Phoenix table in the least convoluted way.
So far I have only found this solution: kafka-to-phoenix, which requires some deep ad-hoc engineering (to my noob eyes). I can tailor the linked solution to my needs without major issues, but I wonder if there are better solutions out there.
Basically, I arrive at this point in my code:
streamingData
.writeStream
.option("checkpointLocation", checkpointLocation)
.trigger(Trigger.ProcessingTime(triggerInterval, TimeUnit.MILLISECONDS))
.foreach(
howToInsertToPhoenix?()
)
.start()
.awaitTermination()
I have a running solution dumping the data to HBase, but it's not really less convoluted with the proposed Phoenix one, so it's not easily/promptly generalizable for similar use cases.
Many thanks in advance :)