5

I would like to expose the data table from my oracle database and expose into apache kafka. is it technicaly possible? As well i need to stream data change from my oracle table and notify it to Kafka. do you know good documentation of this use case? thanks

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
promachos4
  • 506
  • 1
  • 5
  • 8
  • 1
    If you simply want to read (select query) and push to Kafka, simple JDBC code is enough. But IF you looking to stream CDC (change data capture ) in real time then you need some kind of product that can read CDC using Logminer or xstream. Striim (I work for Striim by the way) has such product. There are other products too, but no free or open sources that I am aware of. Oracle's Golden Gate can read GG trail to kafka, but not sure it can write CDC too. Hope this is helpful. – PGK Feb 28 '17 at 18:34
  • looks like Slim below already replied more or less same. – PGK Feb 28 '17 at 18:41
  • This question has already been answered: https://stackoverflow.com/questions/29929205/how-to-integrate-oracle-and-kafka – Adam Leszczyński May 26 '20 at 05:16
  • Does this answer your question? [How to integrate Oracle and Kafka](https://stackoverflow.com/questions/29929205/how-to-integrate-oracle-and-kafka) – Adam Leszczyński May 26 '20 at 05:16

4 Answers4

9

  1. You need Kafka Connect JDBC source connector to load data from your Oracle database. There is an open source bundled connector from Confluent. It has been packaged and tested with the rest of the Confluent Platform, including the schema registry. Using this connector is as easy as writing a simple connector configuration and starting a standalone Kafka Connect process or making a REST request to a Kafka Connect cluster. Documentation for this connector can be found here

  2. To move change data in real-time from Oracle transactional databases to Kafka you need to first use a Change Data Capture (CDC) proprietary tool which requires purchasing a commercial license such as Oracle’s Golden Gate, Attunity Replicate, Dbvisit Replicate or Striim. Then, you can leverage the Kafka Connect connectors that they all provide. They are all listed here

  3. Debezium, an open source CDC tool from Redhat, is planning to work on a connector that is not relying on Oracle Golden Gate license. The related JIRA is here.

Slim Baltagi
  • 129
  • 1
  • Hi Slim, that is a comprehensive answer. Only thing I want to add here is Striim can read CDC in real time and write to a kafka in real time with fault tolerance. (As I mentioned above, I work for Striim, so I knew...-:) ). – PGK Feb 28 '17 at 18:48
  • As of 2018, dbvisit is dead, see e.g. https://rmoff.net/2018/12/12/streaming-data-from-oracle-into-kafka-december-2018/ which looks like a well informed post generally. I think it's interesting that dbvisit successfully does Oracle replication so they do know how to read redo logs, but it looks like it takes more than that. Be careful around self build trigger solutions relying on timestamps since they are vulnerable to race conditions and errors which ruin the order. – ynux Jun 14 '19 at 13:12
0

You can use Kafka Connect for data import/export to Kafka. Using Kafka Connect is quite simple, because there is no need to write code. You just need to configure your connector.

You would only need to write code, if no connector is available and you want to provide your own connector. There are already 50+ connectors available.

There is a connector ("Golden Gate") for Oracle from Confluent Inc: https://www.confluent.io/product/connectors/

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
0

At the surface this is technically feasible. However, understand that the question has implications on downstream applications. So to comprehensively address the original question regarding technical feasibility, bear in mind the following:

  1. Are ordering/commit semantics important? Particularly across tables.
  2. Are continued table changes across instance crashes (Kafka/CDC components) important?
  3. When the table definition changes - do you expect the application to continue working, or will resort to planned change control?
  4. Will you want to move partial subsets of data?
  5. What datatypes need to be supported? e.g. Nested table support etc.
  6. Will you need to handle compressed logical changes - e.g. on update/delete operations? How will you address this on the consumer side?
Brandon Minnick
  • 13,342
  • 15
  • 65
  • 123
Alok Pareek
  • 1
0

You can consider also using OpenLogReplicator. This is a new open source tool which reads Oracle database redo logs and sends messages to Kafka. Since it is written in C++ it has a very low latency like around 10ms and yet a relatively high throughput ratio.

It is in an early stage of development but there is already a working version. You can try to make a POC and check yourself how it works.

Adam Leszczyński
  • 1,079
  • 7
  • 13