0

We are trying to measure latency of message writes in Infosphere CDC- Confluent kafka setup. We have dummy table and dummy transactions will be performed.

Dataflow- MF DB2 table --> IIDR --> Kafka topic

(For Kafka,the producer is iidr)

What could be the right way to find out write latency of sample transactions without any enterprise monitoring tool? Can we check the logs for writes and consider each transaction timestamp and compare with cdc transaction logs to find out the write latency?

PS- I have jconsole configured to collect kafka metrics. But how can we find write latency of a particular CDC stream to kafka topic?

Thanks!

Tim Post
  • 33,371
  • 15
  • 110
  • 174
Tony
  • 671
  • 1
  • 9
  • 29
  • What do you mean by latency of sample transactions? In Kafka or whole transaction? If you just need latency in Kafka, take a look at [this](https://medium.com/@danielmrosa/monitoring-kafka-b97d2d5a5434) – Amin Mar 05 '19 at 07:40
  • It means once we perform some transactions (say 50) at DB2 source, what time it takes to get write acks in Kafka? The data flow would be db2-->IIDR(CDC)-->Kafka. – Tony Mar 05 '19 at 09:18
  • Ideally,we would want latency of whole transaction. But even if we find out latency in kafka,we can estimate the whole transaction latency with the help of response times in IIDR(CDC) logs. – Tony Mar 05 '19 at 09:20

1 Answers1

0

(FYI: I work for IBM and for the IDR CDC Product Specifically)

The integrated KCOPs (Kafka Custom Operation Processors) in the IDR Replication product do not supply a timestamp argument when creating the Kafka producer records. This is also the case if you use the default behaviour without a KCOP.

As such Kafka will generate a timestamp for each producer record when it receives it and associate it with the record. This timestamp value is included with the consumer record when the message is read.

This can be compared with a time generated from the source, by subtraction. There is a timestamp journal control field which can be replicated with the row data. On some sources, Db2 LUW for instance, it represents the commit time of the transaction. On some represents the time that particular row was added to the log.

https://www.ibm.com/support/knowledgecenter/en/SSTRGZ_11.4.0/com.ibm.cdcdoc.mcadminguide.doc/refs/recordmodificationtime.html

Generally I would argue that latency is best considered from the time the commit actually takes effect, but it depends on what you are looking to understand.

I believe on DB2 Z, each one is the time of the operation's log entry, you could take a rough approximation by using the timestamp entry of the last operation in the transaction. This ignores the amount of delay between the writing of the last transaction and the actual commit, but typically these are quite close.

IDR does offer requests for feature enhancement, so please feel free to reach out through either support or this if you'd like to see an enhancement.

Regards,

Shawn
  • 258
  • 1
  • 7