Stack Overflow
Stack Overflow

Questions tagged [confluent-schema-registry]

Schema Registry provides a store, backed by Kafka, for storing and retrieving Avro schemas. It also provides Avro serializers for Kafka clients.

Schema Registry is part of Confluent Open Source Stream Processing Platform.

It provides a serving layer for your metadata. It provides a RESTful interface for storing and retrieving Avro schemas. It stores a versioned history of all schemas, provides multiple compatibility settings and allows evolution of schemas according to the configured compatibility setting. It provides serializers that plug into Kafka clients that handle schema storage and retrieval for Kafka messages that are sent in the Avro format.

1076 questions
2
votes
0 answers

Spark structured streaming how to write to Kafka in Protobuf format

Spark: 3.0.0 Scala: 2.12 confluent I am having spark structured streaming job and looking for an example for writing data frames to Kafka in Protbuf format. I read messages from PostgreSQL and after doing all the transformations have a data frame…
JDev
  • 1,662
  • 4
  • 25
  • 55
2
votes
2 answers

How do you configure the subject naming strategy for Confluent Schema Registry?

I'm using Confluent Schema Registry with Avro and I'd like to use multiple schemas in one Kafka Topic. The default subject naming strategy TopicNameStrategy does not allow for this because it couples the schema subject name to the topic name. It is…
Jared Petersen
  • 125
  • 3
  • 12
2
votes
1 answer

Does it make sense to use the schema registry with Avro SpecificRecord?

I am a consumer and want to use SpecificRecord in Avro for type safety to avoid manually mapping things from a GenericRecord. As I understand things it makes little sense for me to still integrate with the schema registry because I could not do…
Fleshgrinder
  • 15,703
  • 4
  • 47
  • 56
2
votes
1 answer

How to use avro native decoder when using a PollableMessageSource input in spring cloud stream?

I'm using a PollableMessageSource input to read from a Kafka topic. Messages on that topic are in Avro. use-native-decoding was set to true when those messages were published. This is how I'm…
Milad
  • 608
  • 1
  • 6
  • 14
2
votes
1 answer

Confluent cloud | Java Avro Serialization | junk characters in value

I am trying with basic Avro publisher in java which is available in examples https://github.com/confluentinc/examples/blob/5.5.1-post/clients/avro/src/main/java/io/confluent/examples/clients/basicavro/ProducerExample.java I was successfully able to…
Shivakumar ss
  • 653
  • 7
  • 19
2
votes
2 answers

No Schema Validation on Confluent Cloud

I am using confluent cloud and currently i want to prevent messages with bad schema to ingest in my kafka topic. For that i am using schema management feature of confluent where i want to validate if the message is not matching the specified schema…
vicky
  • 561
  • 1
  • 7
  • 17
2
votes
1 answer

How to use SFTP CSV SOURCE connector in kafka using key based authentication?

I am trying to use confluent Kafka SFTP CSV SOURCE Connector. In this I need to authenticate that SFTP SERVER by using private key. I don't know how to work with tls.private.key! How can I achieve this? Sample Configuration: { "tasks.max":…
ROCKY
  • 21
  • 1
2
votes
2 answers

two completely different schema-registry for same broker cluster

I am running two schema registry for same kafka broker cluster with different kafkastore.topic on different machine. kafkastore.topic=_DEV_schemas kafkastore.topic=_QA_schemas The instance comes without any issues and runs fine. But the problem…
Senthil13
  • 79
  • 7
2
votes
0 answers

How do you write to Kafka using the ABRiS library in PySpark?

Has anyone been able to write to Kafka using this library using PySpark? I've been able to successfully read using the code from the README documentation: import logging, traceback import requests from pyspark.sql import Column from…
Brandon
  • 375
  • 2
  • 16
2
votes
0 answers

How to auto-save Avro schema in Confluent Schema Registry from Apache NiFi flow?

How to auto-save Avro schema in Confluent Schema Registry from Apache NiFi flow? That's basically the question. I am not finding the way of automatically storing the Avro schema of the record in the Confluent Schema Registry from a NiFi flow. It is…
Sergey Shcherbakov
  • 4,534
  • 4
  • 40
  • 65
2
votes
1 answer

KafkaStreams, Error Registering Avro Schema

Disclaimer: My experience with KafkaStreams is quite limited. I do not quite understand why am I getting a org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: and Schema being registered is incompatible with an…
Ivo
  • 450
  • 3
  • 18
2
votes
2 answers

docker schema registry and kafka rest cannot up

both zookeeper and Kafka communicated well and up running. not sure why the schema registry and Kafka rest cannot up. below is the docker-compose file. this is the schema and rest docker-compose file. error on this docker [main] ERROR…
user2201789
  • 1,083
  • 2
  • 20
  • 45
2
votes
1 answer

Unable to register schema using register() in python

I am trying to register schema to confluent schema registry using python. from schema_registry.client import SchemaRegistryClient subject_name = "new-schema" schema_url = "https://{{ schemaRegistry }}:8081" sr =…
1689
  • 117
  • 1
  • 10
2
votes
1 answer

serialize Kafka messages with confluent registry under Flink 1.9.1

is it possible to publish message to Kafka serialized with KafkaAvroSerializer by Confluent. I'm using Flink 1.9.1 have saw that some development is going on newer version of flink-avro (1.11.0) but I’m stick to the version. I would like to use the…
imalik8088
  • 1,501
  • 5
  • 21
  • 39
2
votes
1 answer

how to evolve schema in confluent schema registry

I'm working with Avro/Kafka and Confluent's Schema Registry for Avro. I made some basic schemas and subjects with basic types using avsc files and avdl. I am looking at the API's documentation made by Confluent to try to evolve a Schema to version…
WilsonPena
  • 1,451
  • 2
  • 18
  • 37
1 2 3
…
71 72