0

Lets say you have a business Application producing and storing Enriched Product Master Data in its own environment, one the enrichment is completed you want to make that data available on a CouchBase Database.

In order to get that data from Business Application's environment into CouchBase, let's assume I want to use Kafka to broadcast the changes and Nifi to distribute it to the final desination (CouchBase).

But CouchBase takes JSON format files. Can I use Kafka or Nifi to convert the pulled data into a JSON format? I know I can for instance put solution such as Attunity between the Business Application and Kafka to replicate the data real time. But let us assume that there is no budget to implement the solution attunity, so one will temporarily use a REST API on the Business Application side and pull that data with (based on made changes) Kafka, can I convert the data into JSON with Kafka? or NiFi?

EDIT:

Well the reason why I want to know if NiFi can do this, is because our landscape is a bit more complex than I described. Because between CouchBase and the Business Application, you have: [Business App] - [ X ] - [Kafka] - [NiFi] - [DC/OS with KONG API Layer] - [CouchBase Cluster].

And I want to know if I should implement a new solution for Data Replication on the spot of the X, or should I just make use of the Business App REST API and pull data from the REST API with Kafka and Convert my data to JSON in NiFI.

Dennis Jaheruddin
  • 21,208
  • 8
  • 66
  • 122
MSD
  • 311
  • 2
  • 3
  • 20
  • 1
    Kafka itself does not look at the data in any way. You could have a stream processor that converts data from one topic into another topic as JSON. Or that REST endpoint could transform what it reads on the fly. Either way, it does not seem to be a significant effort (one that requires a "budget"). – Thilo Oct 27 '17 at 10:03
  • So you mean a solution such as Apache NiFi could convert the obtained data into JSON ? or the REST API on the Business Applicaiton side? – MSD Oct 27 '17 at 10:04
  • 1
    Kafka does not care about the message format. It has no "official payload format". Transformations (from what? Kafka would not know about that either) to JSON are done outside of Kafka. If you want to do that in some middleware product or in application code is up to you. – Thilo Oct 27 '17 at 10:07
  • @Thilo ; see my EDIT please. – MSD Oct 27 '17 at 12:03

1 Answers1

0

There is a Couchbase sink for Kafka Connect. This will enable you to do exactly what you want. Simple configuration-file based approach.

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92