0

Need help, below code is working fine when I run it on Linux(VM) system but the same code if run on Kubernetes it throws an error.

I am using same certificate file both on Linux and Kubernetes system. I validated, file is present on both locations, but code is failing to run over other.

In fact when I was running this code on AirFlow, I was getting same error so moved to other system and getting exactly same error again.

For some reason it is NOT reading file I think.

  • confluent-kafka version=2.0.2

Please advise!
Error Message:

Traceback (most recent call last):
File "/apps/code/kafka_read_msgs.py", line 20, in consumer = Consumer(conf) cimpl.KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create consumer: ssl.ca.location failed: error:05880002:x509 certificate routines::system lib"}

This is my code:

import os, json, ssl
from confluent_kafka import Consumer, KafkaError, TopicPartition
import time

os.environ['SSL_CERT_FILE'] = "/apps/files/ca.crt"

conf = {
    'bootstrap.servers': 'bootstrap.com:443',
    'group.id': 'xyz',
    'auto.offset.reset': 'earliest',
    'enable.auto.commit': False,
    'ssl.ca.location': '/apps/files/ca.crt',
    'sasl.mechanism':'SCRAM-SHA-512',
    'sasl.username':'user',
    'sasl.password':'pwd',
    'security.protocol':'SASL_SSL',
    'api.version.request': True
}

consumer = Consumer(conf)

def cnt_consumer_msg():
    tot_msgs=0
    tot_module_msgs=0
    
    consumer.subscribe(['topic.dataload'])
    while True:
        msg = consumer.poll(1.0)
        if msg is None:
            print("No more Kafka messages to COUNT")
            print(f"Total Number of messages  are: {tot_msgs}")
            break
        elif msg.error():
            print(f'Error returned by poll: {msg.error()}')
            continue
        else:
            tot_msgs+=1
    print("Number of Metadata Messages uploaded into Kafka: ", tot_msgs)
    return tot_msgs

cnt=cnt_consumer_msg()

print("Number of messages are :",cnt)
zx485
  • 28,498
  • 28
  • 50
  • 59
AnumNuma
  • 31
  • 3

1 Answers1

0

It is saying the /apps/files/ca.crt is not found.

Check the path you're in while executing the code.

For example if you have src/apps/files/ca.crt, but you're executing the code not within src you'll get this error.

Check the path you're in when executing that code in order to adjust the path. You can do:

import os
print(os.getcwd())

this will return the path you're in while executing the code

Bradypodion
  • 51
  • 6