0

I'm trying to use hive-jdbc-uber-jar and configure the JDBC sink connector.

But the connector is throwing error:

[2022-08-31 00:21:21,583] INFO Unable to connect to database on attempt 1/3. Will retry in 10000 ms. (io.confluent.connect.jdbc.util.CachedConnectionProvider)
java.sql.SQLException: **No suitable driver** found for **jdbc:hive2**://XX.XX.XX.XX:10002/test;auth=noSasl;transportMode=http;httpPath=cliservice

Config

    {
      "name": "connector_schema_test_v01",
      "config": {
        "value.converter.schema.registry.url": "http://schema-registry:8081",
        "key.converter.schema.registry.url": "http://schema-registry:8081",
        "name": "connector_schema_test_v01",
        "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
        "tasks.max": "1",
        "key.converter": "io.confluent.connect.avro.AvroConverter",
        "value.converter": "io.confluent.connect.avro.AvroConverter",
        "topics": "topic_schema_test_v05",
        "connection.url": "jdbc:hive2://XX.XX.XX.XX:10002/test;auth=noSasl;transportMode=http;httpPath=cliservice",
        "connection.user": "",
        "connection.password": "",
        "insert.mode": "upsert",
        "delete.enabled": "true",
        "table.name.format": "delta_try_v1_2",
        "pk.mode": "record_key",
        "pk.fields": "K1, K2",
        "auto.create": "true",
        "auto.evolve": "true"
      }
    }

The driver is in the path: /usr/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib

I have restart the connector, but same error. I think driver class name has to be set in some property.

The driver and the URL are working in a SQL editor enter image description here

Any idea?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Luis Estrada
  • 371
  • 7
  • 20
  • are you trying to export data into Delta Lake table? – Alex Ott Aug 31 '22 at 07:54
  • yes, Delta Lake is integrated with apache spark – Luis Estrada Aug 31 '22 at 11:38
  • The jdbc driver is not a Connect plugin. It needs to be on the JVM classpath, such as Kafka libs folder. But also, if you're using Spark, you might have better luck with Structured Streaming from Kafka, then using Delta libraries to write from there... Not use Hive at all – OneCricketeer Aug 31 '22 at 13:45

2 Answers2

1

The jdbc sink connector does not have the HIVE dialect implemented so you cannot use the connector

https://github.com/confluentinc/kafka-connect-jdbc/tree/master/src/main/java/io/confluent/connect/jdbc/dialect

Mar Estrada
  • 41
  • 5
0

The hive-jdbc-uber-jar project was created in 2014 and now comes with this warning:

When I first created this project in 2014 the Hive project did not produce a "standalone" jar that reliably contained all required dependencies to successfully create a JDBC connection. Since that time the community has resolved many, if not all, of those early issues. As of today, the "standalone" jar published by recent versions of Hive make this project mostly obsolete. You can grab the official Hive standalone jar using maven.

Perhaps you can just use Hive as follows and that will solve your issue:

<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-jdbc</artifactId>
    <version>YOUR VERSION OF HIVE</version>
    <classifier>standalone</classifier>
</dependency>
Powers
  • 18,150
  • 10
  • 103
  • 108
  • thanks, I already tried using latest hive-jdbc jar. I beleave the issue is with upsert, using a SQL editor with hive-jar and hive-jdbc, I can run select * from table (non delta, for delta lake tables it doesn't work). But update queries dont work – Luis Estrada Sep 07 '22 at 12:13
  • @LuisEstrada - Did you add the Delta Lake jar as well? What query engine are you using? – Powers Sep 07 '22 at 12:23
  • Yes. Delta libs: core, sharing-spark and storage. I guess the query is thru hive-jdbc-standalone in DBeaver sql editor – Luis Estrada Sep 07 '22 at 16:45