using confluent kafka-schema-registry-client with basic auth with managed confluent schema registry in databricks

Question

in my spark application I have the following scala code

val restService = new RestService(schemaRegistryUrl)
val props = Map(
  "basic.auth.credentials.source" -> "USER_INFO",
  "basic.auth.user.info" -> "%s:%s".format(key, secret) 
).asJava
    
val schemaRegistryClient = new CachedSchemaRegistryClient(restService, 100, props)
// later:
schemaRegistryClient.getSchemaById(id) // fails with 401

I've verified I'm able to get a schema by the same id, from the rest API using the same basic auth credentials. But with the code, I get an auth error - 401. So obviously I'm missing something here and I'll be glad for help here.

Version of both restService and SchemaRegistry 6.2.1 (the latest I believe)

Note this works locally when I run this code with the same credentials via the same schema registry, it works from databricks when I use python with the same connection variables. just with Scala from databricks I get 401

score 1 · Accepted Answer · answered Oct 21 '21 at 17:52

So the answer turned out to be, my assembly build dropped some important stuff when I dropped META-INF, not dropping it, but choosing wisely what I need took some effort, but worked eventually. and then it worked.

Also turns out that the version of the Confluent schema registry client I've used avro version (1.10) isn't compatible with existing versions of spark in Databricks (3/3.1) which support only 1.8

using confluent kafka-schema-registry-client with basic auth with managed confluent schema registry in databricks

1 Answers1