0

I am looking for a way to connect to Confluent Platform to consume messages into Snowflake directly. I do not believe the Confluent Snowflake Sink Connector will work, as we are using Confluent Platform, not cloud.

I tried importing the confluent-kafka packages into Snowpark via a .zip file in S3, since it's not in Snowpark's Anaconda build, and I keep getting this error on the import statement in Python in Snowpark via the constructor method of this package. AttributeError: module 'os' has no attribute 'add_dll_directory'

I'm not sure if this package is just not able to be imported into Snowpark, or if I am missing something simple. The 3 folders I have in the zip are confluent_kafka, confluent_kafka.libs, confluent_kafka-2.2.0.dist-info

Even if I comment out this line in the constructor, it then starts throwing errors on finding package refs like can't find confluent_kafka.cimpl

Not sure what I'm doing wrong here.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
mameesh
  • 3,651
  • 9
  • 37
  • 47

1 Answers1

0

consume messages into Snowflake directly

Kafka Connectors work with any Kafka installation, and are not limited to Cloud, or even Confluent Platform.

You can download and install from https://www.confluent.io/hub/snowflakeinc/snowflake-kafka-connector

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245