0

With Java Client producer, the message can be fine tune to comply with the schema format before publish to topics.

With kafka rest proxy, how to reject messages if the message unable to be deserialize with kafka avro schema version? This is to prevent junks to be added by clients that not comply to schema. I see that version upgrade automatically with new schema to a topics. How to restrict the messages publish to the topic? It could be due to producer clients of all bugs.

I am searching the document and I am new to learn kafka. I know consumer can be smarter with offset but i want to clean up junks from a topics. Thanks.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Tachyonis Space Github
  • 1

2 Answers2

0

There's no way to prevent this with Apache Kafka without putting some reverse proxy that denies the schema update request in front of the Registry, for example. And you'd have to deny access to anyone bypassing that proxy

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Can you enlightened me where to "put some reverse proxy" to denies the update? I thought with schema_version_id field should allows KAFKA REST PROXY to verify and reject the message that failed with the schema. SSL/TLS can authenticate client but the body of messages need to pass the serialize/de serialize test. – Tachyonis Space Github Nov 23 '19 at 13:42
  • It's an HTTP proxy that would replace the address of your schema registry. It would forward any connections to it. The rest proxy doesn't verify any schemas, AFAIK – OneCricketeer Nov 23 '19 at 16:39
0

If I understood your question correctly, you need to reject all events those are non compatible with your standard schema, if its correct then it would be helpful if you try AVRO schema registry compatibility checks.

How to freeze new version of AVRO schema of a topic so that non compliant message is rejected?

If you want to freeze a version of schema, then use FULL/FULL_TRANSITIVE compatibility type with all required fields, now you cannot delete any field because registry wont allow you to delete non-optional fields, and can only add optional fields. So that you can freeze on removal of existing fields, however optional still can added and this shouldn't be a impact on other non-relevant clients.

Hope it helps!

Amol Fasale
  • 942
  • 1
  • 10
  • 32
  • The schema registry would have to already be used since the REST Proxy doesn't support avro without it – OneCricketeer Nov 22 '19 at 06:50
  • true, updated the comment to focus on relevant things. – Amol Fasale Nov 22 '19 at 08:48
  • You mean like this { "type": "record", "name": "HandshakeResponse", "namespace": "org.apache.avro.ipc", "fields": [ {"name": "match", "type": {"type": [ "enum", "FULL/FULL_TRANSITIVE"], "name": "HandshakeMatch", "symbols": ["BOTH", "CLIENT", "NONE"]}}, {"name": "serverProtocol", "type": ["null", "string", "FULL/FULL_TRANSITIVE"]}, {"name": "serverHash", "type": ["null", "FULL/FULL_TRANSITIVE",{"type": "fixed", "name": "MD5", "size": 16}]}, {"name": "meta", "type": ["null", "FULL/FULL_TRANSITIVE",{"type": "map", "values": "bytes"}]} ] } – Tachyonis Space Github Nov 23 '19 at 13:29