Unable to read data from kafka using GCP Cloud Function

Question

I want to read data from kafka Topic through GCP Cloud Function, for which am importing below packeges

Imports inside my main.py file

from kafka import KafkaConsumer
from confluent_avro import AvroKeyValueSerde, SchemaRegistry
from confluent_avro.schema_registry import HTTPBasicAuth`

my requiremnts.txt file

kafka-python
pandas
confluent_avro
requests

But my test my cloud Function code it give me below error Error:

from confluent_avro import AvroKeyValueSerde, SchemaRegistry
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/confluent_avro/__init__.py", line 8, in <module>
    from confluent_avro.schema_registry import SchemaRegistry
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/confluent_avro/schema_registry/__init__.py", line 1, in <module>
    from confluent_avro.schema_registry import auth, client_http
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/confluent_avro/schema_registry/client_http.py", line 29, in <module>
    RETRY_POLICY = SchemaRegistryRetry(
TypeError: Retry.__init__() got an unexpected keyword argument 'method_whitelist'"

**Can somone please suggest what I am mssing here ? **

I tried with

confluent_avro
confluent_avro~=1.8.0
confluent-avero

but they all are giving me same error.

Is there a specific reason youre using `kafka-python` and not Confluent's Python consumer module? — OneCricketeer, May 18 '23 at 21:04

score 0 · Answer 1 · answered May 31 '23 at 17:01

The issue actually comes from the latest urllib3 version 2.0.0 (2023-04-26) that removes Retry options method_whitelist: https://github.com/urllib3/urllib3/pull/2086

One solution for you is to force urllib3 to latest 1.26.x (on 31/05/2023 the latest is urllib3==1.26.16)

Unable to read data from kafka using GCP Cloud Function

1 Answers1