Kafka connect - Transformers - Blacklist a nested field

Question

Is it possible to remove a nested field using SMT with Kafka connect ?

I know the following works perfectly :

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.blacklist": "FieldFoo"

But this does not work ( assuming the nested field is foo->bar) :

"transforms": "ReplaceField",
"transforms.ReplaceField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.blacklist": "FieldFoo.NestedFieldBar",

My data is in avro format.

I don't want to modify the data itself ( like flattening everything) to be able to do that. Any way ?

OneCricketeer · Accepted Answer · 2022-02-23T17:54:15.140

3

All (or at least most) of the Kafka Connect Transforms only work on top-level fields via .get and .put calls to a Struct or Map<String, ?>

You can inspect the source here -

https://github.com/apache/kafka/blob/2.3/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/ReplaceField.java#L150-L163

I would suggest searching/opening a JIRA (and KIP) because it's a long-outstanding issue, IMO

The issue, though, would be that "FieldFoo.NestedFieldBar" is a valid String in itself, so it's hard to differentiate that between the following objects without extra characters like back-ticks or KSQL-like approach of FieldFoo->NestedFieldBar

"FieldFoo.NestedFieldBar": "value"

and

"FieldFoo" : { 
  "NestedFieldBar": "value"
}

edited Feb 23 '22 at 17:54

answered Oct 31 '19 at 16:10

OneCricketeer

179,855
19
132
245

Can we flatten nested ArrayList? – Vishal Dhanani Feb 23 '22 at 17:51
ArrayList isn't a builtin Connect type, but no, arrays cannot be flattened. – OneCricketeer Feb 23 '22 at 17:53

score 0 · Answer 2 · answered Jan 28 '21 at 11:58

0

You have to use "flatten" transformer first to include a "." (period) within the nested fields. So in your case the below should work.

"transforms": "flatten,ReplaceField",
"transforms.flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value",
"transforms.flatten.delimiter": "."
"transforms.ReplaceField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
"transforms.ReplaceField.blacklist": "FieldFoo.NestedFieldBar"

answered Jan 28 '21 at 11:58

Mahaveer Jangir

597
7
15

The above solution is not working when you have any Arraylist on your data. Do you know how to tackle the Arraylist flatten and black list the Arraylist? – Vishal Dhanani Apr 06 '22 at 15:03

Kafka connect - Transformers - Blacklist a nested field

2 Answers2