Load data from separate kafka cluster to Samza?

Question

I am trying to create a Samza job that as closely resembles the Wikipedia example job as I can make it. However in the "WikipediaFeed" object I am trying to get data from a different Kafka broker than the Kafka broker that is running when you start the Hello-Samza grid.

Do I have to create a thread safe Kafka consumer inside the "WikipediaFeed" object to consume data from a different Kafka cluster or is there another way I'm not seeing?

Edit 1: Here is a link to their Wikipedia example. https://github.com/apache/samza-hello-samza/tree/master/src/main

Thanks

Just to be sure, your question is you want create a Samza to consume 2 different Kafka topic and one Kafka topic is on Cluster 1 and the second is on the cluster 2 ? — MaximeF, Sep 22 '17 at 15:16
Right, so I have a fronting Kafka cluster that get's data pushed to it and a Samza cluster that has it's own internal Kafka instance. I want to pull data from the fronting Kafka cluster into the Samza Kafka instance so I can do work on it. — Gregory Saxton, Sep 22 '17 at 15:29
Your Samza run into Cluster 1 who consume 1, 2 or 10 Kafka topics from cluster 2 right ? — MaximeF, Sep 22 '17 at 15:51
Correct. Cluster 1 is a co-located Samza/Kafka/ZK and wants to consume multiple topics from a separate three node kafka cluster. — Gregory Saxton, Sep 22 '17 at 15:57

MaximeF · Answer 1 · 2017-09-22T16:00:21.003

In your example you need change this config (https://github.com/apache/samza-hello-samza/blob/master/src/main/config/wikipedia-feed.properties) :

systems.kafka.consumer.zookeeper.connect=KAFKA_CLUSTER_FRONTING:2181
systems.kafka.producer.bootstrap.servers=KAFKA_CLUSTER_FRONTING:9092
task.inputs=kafka.topic1,kafka.topic2,kafka.topic3

Change the config with your Fronting Kafka cluster and add your topic in task.inputs separated with ","

Edit: Just to be clear, you can deploy your Samza into a Cluster 1 and consume a Kafka topic from another cluster. You need change the config in your Samza properties.

To see more information : Samza config

Then if you need send your message after process to another Kafka cluster you will need create another system in your config.

See more information : https://samza.apache.org/learn/documentation/0.13/api/overview.html

If I set the kafka system to my fronting cluster will my output streams exist in my Samza instance? — Gregory Saxton, Sep 22 '17 at 15:50

Load data from separate kafka cluster to Samza?

1 Answers1