Python, can't pass variable to KafkaConsumer

Question

I am working on python application which connect to kafka and consume some data.

def main(argv):
       params = parse_arg(argv)
       logging.info("Connecting to topic\t" + params.tasks_topic)
       consumer = KafkaConsumer(params.tasks_topic,
                             group_id='kafkatester',
                             bootstrap_servers=params.kafka.split(','),
                             auto_offset_reset='latest',
                             enable_auto_commit=False,
                             max_poll_records=1,
                             max_poll_interval_ms=18000)
def parse_arg(argv):
    parser = argparse.ArgumentParser()
    parser.add_argument('-k', '--kafka')
    parser.add_argument('-t', '--tasks-topic')
    args = parser.parse_args()
    return AppParams(args.kafka, args.tasks_topic)

Locally everything works fine. But, when I run it in docker I got unexpected results:

08/09/2019 09:54:57 AM Connecting to topic      taskstest
08/09/2019 09:54:57 AM <BrokerConnection node_id=bootstrap-0 host=MySecretIP:9092 <connecting> [IPv4 ('MySecretIP', 9092)]>: connecting to MySecretIP:9092 [('MySecretIP', 9092) IPv4]
08/09/2019 09:54:57 AM Probing node bootstrap-0 broker version
08/09/2019 09:54:57 AM <BrokerConnection node_id=bootstrap-0 host=MySecretIP:9092 <connecting> [IPv4 ('MySecretIP', 9092)]>: Connection complete.
08/09/2019 09:54:57 AM Broker version identifed as 1.0.0
08/09/2019 09:54:57 AM Set configuration api_version=(1, 0, 0) to skip auto check_version requests on startup
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/app/src/main.py", line 40, in <module>
    main(sys.argv[1:])
  File "/app/src/main.py", line 26, in main
    max_poll_interval_ms=18000)
  File "/usr/local/lib/python3.7/site-packages/kafka/consumer/group.py", line 390, in __init__
    self._subscription.subscribe(topics=topics)
  File "/usr/local/lib/python3.7/site-packages/kafka/consumer/subscription_state.py", line 120, in subscribe
    self.change_subscription(topics)
  File "/usr/local/lib/python3.7/site-packages/kafka/consumer/subscription_state.py", line 169, in change_subscription
    self._ensure_valid_topic_name(t)
  File "/usr/local/lib/python3.7/site-packages/kafka/consumer/subscription_state.py", line 142, in _ensure_valid_topic_name
    raise ValueError('Topic name "{0}" is illegal, it contains a character other than ASCII alphanumerics, ".", "_" and "-"'.format(topic))
" is illegal, it contains a character other than ASCII alphanumerics, ".", "_" and "-"

It looks like the KafkaConsumer can't handle the params.tasks_topic variable.. why?

kafka-python version is 1.4.6, and python 3.7.3.

What is the name of the topic (i.e. what is the value assigned to `params.tasks_topic`) ? — Giorgos Myrianthous, Aug 09 '19 at 13:16
Does it work if instead of `params.tasks_topic` you hardcode the topic name? — Giorgos Myrianthous, Aug 09 '19 at 13:28
What is the output of this one? `isascii = lambda s: len(s) == len(s.encode())` and then `print(isascii(params.tasks_topic))` — Giorgos Myrianthous, Aug 09 '19 at 13:35
Looks like you somehow end up passing topic name prepended with a bunch of spaces, like '____taskstest'. Can you try to `trim(params.tasks_topic)`? — mazaneicha, Aug 09 '19 at 18:57
You are right. But, instead `trim` I used `str.strip` - it works. — Kuba Wenta, Aug 11 '19 at 06:38

score 0 · Answer 1 · answered Aug 12 '19 at 18:13

You need to trim the topic Looks like you sending the topic name with space or white spaces So you need first to validate the topic is str. Trimmed str So my suggestion goes like this :

def main(argv):
   params = parse_arg(argv)
   __topic  = params.tasks_topic
   print('before trim : ' + __topic)
   print('after trim : ' + __topic.strip())
   logging.info("Connecting to topic\t" + __topic.strip() )
   consumer = KafkaConsumer( __topic.strip() ,
                         group_id='kafkatester',
                         bootstrap_servers=params.kafka.split(','),
                         auto_offset_reset='latest',
                         enable_auto_commit=False,
                         max_poll_records=1,
                         max_poll_interval_ms=18000)
def parse_arg(argv):
    parser = argparse.ArgumentParser()
    parser.add_argument('-k', '--kafka', default='lola')
    parser.add_argument('-t', '--tasks-topic',default='pola')
    args = parser.parse_args()
    return AppParams(args.kafka, args.tasks_topic

I have tested it works like a charm ;) Ping me if any help is needed ;)

@mazaneicha already gives an answer in the comment above. So in implementation, used `strip()` function inside `parse_arg()` and if I didn't do it I would have to remove whitespaces in every place when I use `tasks_topic`.. — Kuba Wenta, Aug 12 '19 at 18:39

Python, can't pass variable to KafkaConsumer

1 Answers1