4

I am building a performance monitoring tool which works in a cluster with Kafka topics.

For example, I am monitoring two topics: request, response. I.e. I need to have two timestamps - one from request and another from response. Then I could calculate difference to see how much time spent in a service which received a request and produced a response.

Please take in the account that it is working on a cluster, so different components may run on different hosts, hence - different physical clocks - so they could be out-of-sync and it will distort results significantly.

Also, I could not reliably use the clock of the monitoring tool itself, as this will influence timing results by its own processing times.

So, I would like to design a proper way which is reliably calculate time difference. What is most reliable way to measure time difference between two events in Kafka?

kan
  • 28,279
  • 7
  • 71
  • 101
  • So the timestamps are set by the application that produces the messages to the `request` and `response` topics? Or you are using the timestamp of the Kafka message itself? And when you talk about measuring the time between the two events, how are you defining _the time_ of the event? – Robin Moffatt Feb 11 '19 at 13:12
  • @RobinMoffatt Events are produced by different applications. E.g. app A sends a request, app B responses to it, and the monitoring tool listens to both events to find a difference. So, I want to measure how fast app B is. The "time of event" - yes, I am not sure about this. I guess the best definition would be as a difference between when app B could consume `request` event and when it made `response` available for others to consume. In this case it could be Kafka Broker's timestamp I guess. – kan Feb 11 '19 at 13:41
  • You can set `log.message.timestamp.type` to `LogAppendTime` which use broker timestamp in the records. Then, you can find the time diff b/w request and response. – Kamal Chandraprakash Feb 14 '19 at 11:45
  • @KamalChandraprakash How to be sure that `request` and `response` topics are using the same physical clock? My understanding Kafka may work on a several nodes of a cluster... – kan Feb 14 '19 at 14:32

1 Answers1

0

Solution 1:

We had similar problem before and solution we had was setting up NTP ( network time protocol).

In this one of your node act as NTP server and runs demons to keep time in sync across all your nodes we kept UTC and all other nodes has NTP clients which kept same time across all the servers

Solution 2:

Build a clock common API for all your components which will provide current time. This will make your system design independent of node local clock.

Wai Ha Lee
  • 8,598
  • 83
  • 57
  • 92
sandesh dahake
  • 1,028
  • 7
  • 16