I am working with apache beam. My task is to pull data from kafka topic and process in dataflow.
Does dataflow support kafkaIO ?
Which runners are supported for KafkaIO ?
Asked
Active
Viewed 215 times
2
Joseph N
- 540
- 8
- 28
-
was the answer helpful? – Pablo Aug 11 '20 at 23:42
-
@Pablo, conceptually its ok., but practically i'll need to research more about runner_v2. – Joseph N Aug 13 '20 at 04:41
1 Answers
0
Yes. KafkaIO is supported by Dataflow and other major Beam runners.
We have Kafka transforms for both Java and Python SDKs.
Note that Python versions are cross-language transforms and Dataflow requires runner V2.
chamikara
- 1,896
- 1
- 9
- 6
-
Thanks @chamikara, as per apache beam documentation, ' --experiments=use_runner_v2 ' is the parameter we need to pass for using RunnerV2. Is it enough or i need to make changes in my code? sorry i am new to apache beam. – Joseph N Jul 08 '20 at 04:39
-
It should be enough. You also need Java installed in the pipeline from where the job is being submitted. We are adding an example to Beam which should clarify this better. – chamikara Jul 14 '20 at 00:50