Getting error "confluent_kafka.libs are not found" when python programs are converted to exe application using pyinstaller(4.5) and librdkafka 1.7.0

Question

The latest version of librdkafka 1.7.0 when used in a python(python 3.8.11) program on windows 64 bit os works fine but when I convert the program to a exe file and run I am getting the following error

File "produce-simple-avro.py", line 1, in
File "", line 991, in _find_and_load
File "", line 975, in find_and_load_unlocked
File "", line 671, in load_unlocked
File "PyInstaller\loader\pyimod03_importers.py", line 546, in exec_module
File "confluent_kafka_init.py", line 18, in
File "confluent_kafka_init.py", line 9, in _delvewheel_init_patch_16682001180
File "os.py", line 1109, in add_dll_directory
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\Users\a\AppData\Local\Temp\_MEI132722\confluent_kafka.libs'
[13284] Failed to execute script 'produce-simple-avro' due to unhandled exception!

How to reproduce

Intsall python 3.8.11 on windows 10 64 bit os.
pip install confluent-kafka[avro]
create a program with the following code


from confluent_kafka import avro
from confluent_kafka.avro import AvroProducer


value_schema_str = """
{
   "namespace": "my.test",
   "name": "value",
   "type": "record",
   "fields" : [
     {
       "name" : "name",
       "type" : "string"
     }
   ]
}
"""

key_schema_str = """
{
   "namespace": "my.test",
   "name": "key",
   "type": "record",
   "fields" : [
     {
       "name" : "name",
       "type" : "string"
     }
   ]
}
"""

value_schema = avro.loads(value_schema_str)
key_schema = avro.loads(key_schema_str)
value = {"name": "Value"}
key = {"name": "Key"}


def delivery_report(err, msg):
    """ Called once for each message produced to indicate delivery result.
        Triggered by poll() or flush(). """
    if err is not None:
        print('Message delivery failed: {}'.format(err))
    else:
        print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition()))

try:
    avroProducer = AvroProducer({
    'bootstrap.servers': 'localhost:9092',
    'on_delivery': delivery_report,
    'schema.registry.url': 'http://localhost:8081'
    }, default_key_schema=key_schema, default_value_schema=value_schema)
    while True:
        avroProducer.produce(topic='my_topic', value=value, key=key)
        avroProducer.flush()
        time.sleep(3)
except Exception as e:
    print(0)

For converting to exe file i will use ' pyinstaller ' == 4.5,using the below command

pyinstaller --onefile name-of-file-with-above-code.py

score 0 · Answer 1 · answered Mar 10 '22 at 21:32

I was able to get around this by adding confluent_kafka.libs directory to datas. If you're using a spec file for pyinstaller, add something like this when initializing Analysis

datas=[('.venv/Lib/site-packages/confluent_kafka.libs', 'confluent_kafka.libs'),]

This led me to another issue with importing confluent_kafka.admin module. It was handled with

hiddenimports=['confluent_kafka.admin',],

Getting error "confluent_kafka.libs are not found" when python programs are converted to exe application using pyinstaller(4.5) and librdkafka 1.7.0

1 Answers1