Unknown Protocol : local with beam.io.kafka.ReadFromKafka

Question

I want to read from Kafka topic using beam but get the following error. Any hints?

I can consume messages using kafka cli perfectly fine.

RuntimeError: Pipeline unique-job-name_7b885f0f-c8fd-4763-bc1e-96817e714dac failed in state FAILED: java.net.MalformedURLException: unknown protocol: local

Here is my code

import re
import logging
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.io.kafka import ReadFromKafka
from apache_beam.io.kafka import WriteToKafka


pipeline_options = PipelineOptions(
    runner='FlinkRunner',
    project='my-project-id',
    job_name='unique-job-name',
)

consumer_config = {
    'bootstrap.servers': 'kafka.test.svc.cluster.local:9092'
}

def main():
    print(beam.__version__)
    with beam.Pipeline(options=pipeline_options) as p:
        (p | "Read from Kafka" >> ReadFromKafka(consumer_config=consumer_config,
                                                topics=["test"],
                                                with_metadata=True)
           )
    
  
  
  
main()