3

I have started exploring Kafka and Kafka connect recently and did some initial set up . But wanted to explore more on schema registry part .

My schema registry is started now what i should do .

I have a AVRO schema stored in avro_schema.avsc.

here is the schema 

{
  "name": "FSP-AUDIT-EVENT",
  "type": "record",
  "namespace": "com.acme.avro",
  "fields": [
    {
      "name": "ID",
      "type": "string"
    },
    {
      "name": "VERSION",
      "type": "int"
    },
    {
      "name": "ACTION_TYPE",
      "type": "string"
    },
    {
      "name": "EVENT_TYPE",
      "type": "string"
    },
    {
      "name": "CLIENT_ID",
      "type": "string"
    },
    {
      "name": "DETAILS",
      "type": "string"
    },
    {
      "name": "OBJECT_TYPE",
      "type": "string"
    },
    {
      "name": "UTC_DATE_TIME",
      "type": "long"
    },
    {
      "name": "POINT_IN_TIME_PRECISION",
      "type": "string"
    },
    {
      "name": "TIME_ZONE",
      "type": "string"
    },
    {
      "name": "TIMELINE_PRECISION",
      "type": "string"
    },
    {
      "name": "AUDIT_EVENT_TO_UTC_DT",
      "type": [
        "string",
        "null"
      ]
    },
    {
      "name": "AUDIT_EVENT_TO_DATE_PITP",
      "type": "string"
    },
    {
      "name": "AUDIT_EVENT_TO_DATE_TZ",
      "type": "string"
    },
    {
      "name": "AUDIT_EVENT_TO_DATE_TP",
      "type": "string"
    },
    {
      "name": "GROUP_ID",
      "type": "string"
    },
    {
      "name": "OBJECT_DISPLAY_NAME",
      "type": "string"
    },
    {
      "name": "OBJECT_ID",
      "type": [
        "string",
        "null"
      ]
    },
    {
      "name": "USER_DISPLAY_NAME",
      "type": [
        "string",
        "null"
      ]
    },
    {
      "name": "USER_ID",
      "type": "string"
    },
    {
      "name": "PARENT_EVENT_ID",
      "type": [
        "string",
        "null"
      ]
    },
    {
      "name": "NOTES",
      "type": [
        "string",
        "null"
      ]
    },
    {
      "name": "SUMMARY",
      "type": [
        "string",
        "null"
      ]
    }
  ]
}

Is my schema is valid .I converted it online from JSON ? where should i keep this schema file location i am not sure about . Please guide me with the step to follow . I am sending records from Lambda function and from JDBC source both .

So basically how can i enforce AVRO schema and test ? Do i have to change anything in avro-consumer properties file ?

Or is this correct way to register schema 

   ./bin/kafka-avro-console-producer \
                 --broker-list b-3.**:9092,b-**:9092,b-**:9092 --topic AVRO-AUDIT_EVENT \
                 --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"f1","type":"string"}]}'





curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json"     --data '{"schema" : "{\"type\":\"struct\",\"fields\":[{\"type\":\"string\",\"optional\":false,\"field\":\"ID\"},{\"type\":\"string\",\"optional\":true,\"field\":\"VERSION\"},{\"type\":\"string\",\"optional\":true,\"field\":\"ACTION_TYPE\"},{\"type\":\"string\",\"optional\":true,\"field\":\"EVENT_TYPE\"},{\"type\":\"string\",\"optional\":true,\"field\":\"CLIENT_ID\"},{\"type\":\"string\",\"optional\":true,\"field\":\"DETAILS\"},{\"type\":\"string\",\"optional\":true,\"field\":\"OBJECT_TYPE\"},{\"type\":\"string\",\"optional\":true,\"field\":\"UTC_DATE_TIME\"},{\"type\":\"string\",\"optional\":true,\"field\":\"POINT_IN_TIME_PRECISION\"},{\"type\":\"string\",\"optional\":true,\"field\":\"TIME_ZONE\"},{\"type\":\"string\",\"optional\":true,\"field\":\"TIMELINE_PRECISION\"},{\"type\":\"string\",\"optional\":true,\"field\":\"GROUP_ID\"},{\"type\":\"string\",\"optional\":true,\"field\":\"OBJECT_DISPLAY_NAME\"},{\"type\":\"string\",\"optional\":true,\"field\":\"OBJECT_ID\"},{\"type\":\"string\",\"optional\":true,\"field\":\"USER_DISPLAY_NAME\"},{\"type\":\"string\",\"optional\":true,\"field\":\"USER_ID\"},{\"type\":\"string\",\"optional\":true,\"field\":\"PARENT_EVENT_ID\"},{\"type\":\"string\",\"optional\":true,\"field\":\"NOTES\"},{\"type\":\"string\",\"optional\":true,\"field\":\"SUMMARY\"},{\"type\":\"string\",\"optional\":true,\"field\":\"AUDIT_EVENT_TO_UTC_DT\"},{\"type\":\"string\",\"optional\":true,\"field\":\"AUDIT_EVENT_TO_DATE_PITP\"},{\"type\":\"string\",\"optional\":true,\"field\":\"AUDIT_EVENT_TO_DATE_TZ\"},{\"type\":\"string\",\"optional\":true,\"field\":\"AUDIT_EVENT_TO_DATE_TP\"}],\"optional\":false,\"name\":\"test\"}"}' http://localhost:8081/subjects/view/versions

what next i have to do

But when i try to see my schema i get only below 

curl --silent -X GET http://localhost:8081/subjects/AVRO-AUDIT-EVENT/versions/latest

this is the result 

{"subject":"AVRO-AUDIT-EVENT","version":1,"id":161,"schema":"{\"type\":\"string\",\"optional\":false}"}

Why i do not see my full registered schema

Also when i try to delete schema

i get below error 

{"error_code":405,"message":"HTTP 405 Method Not Allowed"

i am not sure if my schema is registered correctly .

Please help me. Thanks in Advance

Atharv Thakur
  • 671
  • 3
  • 21
  • 39

1 Answers1

2

is my schema valid

You can use the REST API of the Registry to try and submit it and see...

where should i keep this schema file location i am not sure about

It's not clear how you're sending messages...

If you actually wrote Kafka producer code, you store it within your code (as a string) or as a resource file.. If using Java, you can instead use the SchemaBuilder class to create the Schema object

You need to rewrite your producer to use Avro Schema and Serializers if you've not already

If we create AVRO schema will it work for Json as well .

Avro is a Binary format, but there is a JSONDecoder for it.

what should be the URL for our AVRO schema properties file ?

It needs to be the IP of your Schema Registry once you figure out how to start it. (with schema-registry-start)

Do i have to change anything in avro-consumer properties file ?

You need to use the Avro Deserializer

is this correct way to register schema

.> /bin/kafka-avro-console-producer \

Not quite. That's how you produce a message with a schema (and you need to use the correct schema). You also must provide --property schema.registry.url

You use the REST API of the Registry to register and verify schemas

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Hi ..Thanks for replying ...As usual its you only who takes Kafka learning to next level for people like us .So My schema registry is running good .But as you said i need to keep it into resource file .Can i know where exactly ?I do not have any code just configs .Second when you say i have to rewrite producer what does that mean in context of JDBC connector ?I have to change that only propeties for that ? – Atharv Thakur Jan 07 '20 at 06:13
  • Also when i read topic from JDBC i still see jason format only – Atharv Thakur Jan 07 '20 at 08:58
  • 1) *JavaScript Object Notation* is not written "jason". 2) You must configure AvroConverter in the connector properties to get Avro data. The source connector is a type of producer. 3) Kafka Connect creates its own schemas, so you don't need to worry about those – OneCricketeer Jan 07 '20 at 09:09
  • So i just have to use property file ..Data from JDBC will be converted by connector into AVRO ?I did config changes and started my schema registry and then connector ...But when data is inserted into JDBC and i tried to read from the topic i miss some data ..All columns are not displaying – Atharv Thakur Jan 07 '20 at 09:31
  • It should pull all columns by default. But if you are using avro console consumer, then it's working, and the problem exists elsewhere – OneCricketeer Jan 07 '20 at 09:50
  • Its not working ..I get this error `org.apache.kafka.common.config.ConfigException: Must provide the Avro schema string in value.schema` ... – Atharv Thakur Jan 07 '20 at 11:01
  • Please see the last part of my question ...I am not able to register avro ..Do i need to download avro converter ? – Atharv Thakur Jan 07 '20 at 11:28
  • That error comes from the Avro console producer. Why are you trying to produce data on your own? And AvroConverter comes with Confluent Platform. If you installed via APT/YUM then you'd have it – OneCricketeer Jan 07 '20 at 14:54
  • Just one more question ...Can i have JSON schema fixed for all messages ?If a message which does not fit to that schema should be discared – Atharv Thakur Jan 08 '20 at 07:57
  • The jdbc source manages schemas on the it's own, not only as json. It should be allowed to add and remove columns. If you have constraints on invalid values, you should be using triggers or embed that logic in your code to prevent data from being added – OneCricketeer Jan 08 '20 at 13:36
  • i am still not able to implement AVRO schema .I have schema i have modified AVRO properties file but still not able to use or star it – Atharv Thakur Jan 10 '20 at 06:54
  • @AtharvThakur Why not? Have you read the quickstart about how to use `kafka-avro-console-producer`? https://docs.confluent.io/3.0.0/quickstart.html – OneCricketeer Jan 10 '20 at 06:56
  • yes i did everything ...Can i update my question with all steps i did ? – Atharv Thakur Jan 10 '20 at 06:59
  • You can, but according to the current post, you have no `value.schema`, like the documentation says to have – OneCricketeer Jan 10 '20 at 07:00
  • Edited my question ..Please help me with step for AVRO . – Atharv Thakur Jan 12 '20 at 03:16
  • I think i am on right track ...Please have a look and Guide a little bit – Atharv Thakur Jan 12 '20 at 05:31
  • You wouldn't use the Producer just to register a schema https://docs.confluent.io/current/schema-registry/using.html#common-sr-usage-examples You applications will fail when sending messages using the Avro serializer that don't match that schema – OneCricketeer Jan 12 '20 at 06:23
  • So i have registered my schema also ..I get verison when i do curl ...Can i post my messages now ? – Atharv Thakur Jan 12 '20 at 10:13
  • Sure. You don't need to ask me for permissions. You can always try on your own – OneCricketeer Jan 12 '20 at 10:56
  • still i am not able to make this work ..when i try to see my schema i get only few parts .Also not able to delete schema.I am not sure if my schema is registered correctly ..I have updated my question with the details as well . – Atharv Thakur Jan 13 '20 at 17:32
  • Looks like you used `kafka-avro-console-producer`, which will register the schema for you, which you should **not** do. And Schema Deletion works fine in Confluent Platform 4.0+. `curl -X DELETE http://registry:8081/subjects/name` – OneCricketeer Jan 13 '20 at 21:02
  • No i have used `curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"schema" : ` to register schema – Atharv Thakur Jan 14 '20 at 03:29
  • And Kafka Avro console producer does the same thing internally. If you provided the correct schema, that does not answer how you got something different in the registry – OneCricketeer Jan 14 '20 at 04:00
  • so then what should be correct way to register ...Can you please have link here ..Cai i follow the same thing ..He ha schema aslo in his avro folder https://github.com/datastax/kafka-examples/blob/master/producers/src/main/java/avro/avro_example.sh – Atharv Thakur Jan 14 '20 at 05:11
  • using curl (or POSTMan) is the proper way to register, as you've already said you did. Putting schemas in your code allows you to write a producer, not register them – OneCricketeer Jan 14 '20 at 05:40
  • So i have to register and then i have to use that in code as well ..Or can i write producer code without schema ? – Atharv Thakur Jan 14 '20 at 05:50
  • Avro requires a Schema. You can have the producer use that schema and do a registration on its own, **but** it is highly recommended to actually POST the schema yourself – OneCricketeer Jan 14 '20 at 05:51