8

I need to read from a Kafka feed into a SQL Server database and i'm struggling to find any detail on how to do it. I realise it may be simple, but I've searched and found only stuff about feeding into Kafka from SQL

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
rosscv1
  • 89
  • 1
  • 2
  • Do you know how to read from Kafka? It should just be a matter of executing SQL Server `INSERT` statements for each message. – Dan Guzman Nov 05 '18 at 11:40
  • 2
    @Dan doing a manual consumer loop from Kafka requires careful topic offset management and database error handling/detection. In general, the JDBC Sink Kafka Connect tasks tries their best to handle those, but yes, it just runs a batch of inserts – OneCricketeer Nov 05 '18 at 12:45
  • +1 to @cricket_007's comment. Hand-coding integration between Kafka and RDBMS (and other systems for that matter) is generally a bad idea, since Kafka Connect exists specifically for this purpose and does it better ;) – Robin Moffatt Nov 05 '18 at 14:18
  • This question is not too broad. It very specific, how to read data into apache kafka from SQL server. If you were any more specific then you would probably already know the answer to the question. Please reopen this question. – PrestonDocks Jul 21 '19 at 13:33

1 Answers1

8

Use Kafka Connect, and the JDBC Sink connector.

Kafka Connect is part of Apache Kafka, and provides streaming integration between both sources into Kafka, and from Kafka out to targets. To use it you just set up configuration files describing the source and target of data. You can read more about it in this series of blogs:

This page includes an example of setting up the JDBC sink.

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92