Stack Overflow
Stack Overflow

Questions tagged [confluent-schema-registry]

Schema Registry provides a store, backed by Kafka, for storing and retrieving Avro schemas. It also provides Avro serializers for Kafka clients.

Schema Registry is part of Confluent Open Source Stream Processing Platform.

It provides a serving layer for your metadata. It provides a RESTful interface for storing and retrieving Avro schemas. It stores a versioned history of all schemas, provides multiple compatibility settings and allows evolution of schemas according to the configured compatibility setting. It provides serializers that plug into Kafka clients that handle schema storage and retrieval for Kafka messages that are sent in the Avro format.

1076 questions
3
votes
1 answer

How to get schema along with kafka topic information from kafka client?

I can see that in the java client we can get a list of topics and their configs using Admin.describeTopics(Collection). Is there a way to also get the schemas back from that call? I am new to kafka and not sure how to get the topic information and…
user
  • 352
  • 3
  • 13
3
votes
3 answers

Difference Avro Vs Cloudevent Vs AsyncAPI | Best fit for Schema evolution and naming convention in kafka

I am using confluent schema registry. What is the difference between Avro Vs Cloudevent Vs AsyncAPI What is the best fit for Schema evolution and naming convention in kafka ? Can we use different schema types for different topics based on…
VinuBibin
  • 679
  • 8
  • 19
3
votes
0 answers

In windows cannot start schema registry from confluent

I started zookeeper and kafka from the downloaded package here: http://mirror.klaus-uwe.me/apache/kafka/2.4.0/kafka_2.13-2.4.0.tgz Then I tried with this: https://github.com/confluentinc/schema-registry 1.in one command prompt I start zookeeper…
Teo
  • 330
  • 2
  • 9
3
votes
3 answers

How to programmatically get schema from confluent schema registry in Python

As of now i am doing something like this reading avsc file to get schema value_schema = avro.load('client.avsc') can i do something to get schema from confluent schema registry using topic-name? i found one way but didn't figure out how to use…
Mohit Singh
  • 401
  • 1
  • 10
  • 30
3
votes
3 answers

Differentiating an AVRO union type

I'm consuming Avro serialized messages from Kafka using the "automatic" deserializer like: props.put( ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "io.confluent.kafka.serializers.KafkaAvroDeserializer" ); props.put("schema.registry.url",…
Lee Hambley
  • 6,270
  • 5
  • 49
  • 81
3
votes
1 answer

How to use Schema registry for Kafka Connect AVRO

I have started exploring Kafka and Kafka connect recently and did some initial set up . But wanted to explore more on schema registry part . My schema registry is started now what i should do . I have a AVRO schema stored in avro_schema.avsc. here…
Atharv Thakur
  • 671
  • 3
  • 21
  • 39
3
votes
0 answers

Kafka Schema Registry failed to initialize after ZK and Brokers are configured with SASL_PLAINTEXT Security

We are using Confluent community edition setup for Kafka, currently we have a requirement to configure ACLs around the cluster, accordingly we have configured the zk and broker nodes so clients requires authentication(username/password)…
Amol Fasale
  • 942
  • 1
  • 10
  • 32
3
votes
1 answer

Message value in Confluent Cloud not decoding properly

I'm very new to kafka and confluent. I wrote a Producer nearly identical to the tutorial on https://www.confluent.fr/blog/schema-registry-avro-in-spring-boot-application-tutorial/ with my own dummy model. The application.yaml is the same as well.…
Prady
  • 165
  • 1
  • 15
3
votes
0 answers

Getting Secure Kafka and Schema registry to talk to each other, as well as spring boot configuration for it

Hey I have been trying to solve this for about two weeks now. Basically I want kafka to be on SSL and schema registry on HTTPS. No kerberos to be used. I have a two spring services, one is a producer and the other a consumer (avro) this is my…
Przemek Hendel
  • 60
  • 1
  • 10
3
votes
1 answer

How Kafka Streams API get correct schema from Schema Registry?

I try to understand how Kafka Streams API works with Schema Registry. I know that you must specify Schema Registry URL when you setup your application, but I cannot understand how my application retrieves correct schema from registry without…
Marco Catalano
  • 199
  • 14
3
votes
1 answer

how to ignore the bad message in kafka jdbc sink connector?

I'm using kafka JDBC connector to push data from kafka topic to postgres on aws. However, when a bad message occurs, for example, the data type in the message is different from what is defined in the database, I get error message: ERROR…
Sophie
  • 119
  • 2
  • 6
3
votes
1 answer

Kafka Connect Schema evolution when columns are removed

Lets say we have a setup as follows. Schema evolution compatibility is set to BACKWARD. JDBC Source Connector polls data from DB writing to Kafka topic.HDFS Sink Connector read message from Kafka topic and write to HDFS in Avro format. Following…
Ashika Umanga Umagiliya
  • 8,988
  • 28
  • 102
  • 185
3
votes
1 answer

KafkaStreamsStateStore not working when the store value is an Avro SpecificRecord

I have a Spring Cloud Kafka Streams application that uses a StateStore in the Processor API, when using a transformer to perform a deduplication. The state store key-value are of the following types: . When running the…
codependent
  • 23,193
  • 31
  • 166
  • 308
3
votes
0 answers

Caused by: org.apache.avro.AvroRuntimeException: Unknown datum type org.joda.time.DateTime:

The following field is causing my Kafka Stream application to fail when it tries to produce message. { "name" : "TS", "type" : { "type" : "long", "logicalType" : "timestamp-millis" }, "doc" : "date time indicating the…
user482963
  • 342
  • 6
  • 17
3
votes
1 answer

NPE while deserializing avro messages in kafka streams

I wrote a small java class to test the consumption of Avro encoded Kafka topic. Properties appProps = new Properties(); appProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "http://***kfk14bro1.lc:9092"); …
Nikhil Sahu
  • 2,463
  • 2
  • 32
  • 48
1 2 3
…
71 72