0

So I've been experimenting with Kafka and I am trying to manipulate/change the offsets of the source database using this link https://debezium.io/documentation/faq/. I was successfully able to do it but I was wondering how I would do this in native kafka commands instead of using kafkacat.
So these are the kafka commands that I'm using

kafkacat -b kafka:9092 -C -t connect_offsets -f 'Partition(%p) %k %s\n'

and

echo '["In-house",{"server":"optimus-prime"}]|{"ts_sec":1657643280,"file":"mysql-bin.000200"","pos":2136,"row":1,"server_id":223344,"event":2}' | \
kafkacat -P -b kafka:9092 -t connect_offsets -K \| -p 2

It basically reverts the offset of the source system back to a previous binlog and I can be able to read the db from a previous point in time. So this works well, but was wondering what I would need to compose via native kafka since we don't have kafkacat on our dev/prod servers although I do see it's value and maybe that will be installed later in the future. This is what I have so far for the transalation but it's not quite doing what I'm thinking.

./kafka-console-consumer.sh   --bootstrap-server kafka:9092   --topic 
connect_offsets   --property print.offset=true   --property print.partition=true   -- 
property print.headers=true   --property print.timestamp=true   --property 
print.key=true   --from-beginning

After I run this I get these results. enter image description here

This works well for the kafka consumer command but when I try to translate the producer command I run into issues.

./kafka-console-producer.sh   --bootstrap-server kafka:9092   --topic connect_offsets  
--property parse.partition=true    --property parse.key=true --property 
key.separator=":"

I get a prompt after the producer command and I enter this

["In-house",{"server":"optimus-prime"}]:{"ts_sec":1657643280,"file":"mysql-bin.000200","pos":2136,"row":1,"server_id":223344,"event":2}:2

But it seems like it's not taking the command because the bin log position doesn't update after I run the consumer command again. Any ideas? Let me know.

EDIT: After applying OneCricketeer's changes I'm getting this stack trace. enter image description here

Alex
  • 211
  • 1
  • 11

1 Answers1

1

key.separator=":" looks like it will be an issue considering it will split your data at ["In-house",{"server":

So, basically you produced a bad event into the topic, and maybe broke the connector...

If you want to literally use the same command, keep your key separator as |, or any other character that will not be part of the key.

Also, parse.partition isn't a property that is used, so you should remove :2 at the end... I'm not even sure kafka-console-producer can target a specific partition.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • getting this response now at kafka.tools.ConsoleProducer$LineMessageReader.readMessage(ConsoleProducer.scala:292) at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:51) at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala) – Alex Jul 18 '22 at 17:15
  • Can you edit the question with the full stacktrace? – OneCricketeer Jul 18 '22 at 17:18
  • done thanks! before I changed it to a pipe | I actually didn't get any errors. – Alex Jul 18 '22 at 17:20
  • wait so wait how does it know which partition to locate if I eliminate the 2 ? – Alex Jul 18 '22 at 17:21
  • Read the error. `No key found on line`... You're still using `:` between your key and value. And see answer edit, you cannot have `:2` at the end – OneCricketeer Jul 18 '22 at 17:21
  • You'd need to write your own client to target a specific partition. Otherwise, it'll use the `DefaultPartitioner` logic – OneCricketeer Jul 18 '22 at 17:22
  • oh geez how did I miss the no key found on line :( haha – Alex Jul 18 '22 at 17:22
  • sorry OneCricketeer still just learning but how would one write my own client to target a specific partition, because this would need to be targeted pretty well, because if I just use default it may not be what I want and corrupt my connector. – Alex Jul 18 '22 at 17:24
  • 1
    Pick a programming language you are comfortable with, then build a Kafka producer client in it. Or, just continue using `kcat` – OneCricketeer Jul 18 '22 at 17:39
  • 1
    thanks OneCricketeer I believe avoiding kafkacat may be more trouble in this case :D – Alex Jul 18 '22 at 17:43