0

I am trying to downstream data from kafka topics to a relational db like postgres. I don't want to use kafka connect or jdbc connectors. Is their any way I can use raw queries to export data from kafka to postgres.

I will be using python to implement this

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
androidbear
  • 9
  • 1
  • 4
  • You can, sure. What issues are you experiencing? – OneCricketeer Oct 07 '20 at 21:53
  • I do not know how to do that. Should I use kSQL to query data and then insert it manually in postgres? – androidbear Oct 08 '20 at 11:15
  • Your question asked nothing about querying, only using python, Kafka, and Postgres. You consume a topic, parse fields out, and write to a table... Kafka Connect comes with Kafka, so it's not really clear why you wouldn't use it – OneCricketeer Oct 08 '20 at 14:00
  • I am trying to learn to do it without kafka connect. Do I just parse the event in consumer and use an insert query for postgres? – androidbear Oct 09 '20 at 11:22
  • That's correct. But you also need to handle offset commits, consumer group coordination, restarts, bad data, and any potential table column modification – OneCricketeer Oct 09 '20 at 12:38

0 Answers0