0

I am using spring cloud streams for consuming kafka. I need to get the time the event has been published to the queue from the publisher(publisher time not the broker ingestion time)

I can see below information in headers:

kafka_timestamptype=createtime,
kafka_receivedTopic=Topic_Name,
kafka_receivedTimestamp= 1563108979621,
timetstamp= 1563108984514

I am really confused on above two timestamps. I did see in some sources saying that create time timestamp means that its the publishing timestamp from producer.But what represents the publishing time , is it kafka_ receivedTimestamp or just timestamp?

When using spring cloud streams I noticed that spring message headers has timestamp in springs control. Does that mean kafka_receivedTimestamp is the publishing time of the record.

Documention just says that, “The header for holding the timestamp of the consumer record. “. It doesn’t clarify if its the consuminng time or publishing time or broker ingestion time.

https://docs.spring.io/spring-kafka/api/org/springframework/kafka/support/KafkaHeaders.html#RECEIVED_TIMESTAMP

Could anyone suggest what does these two timestamps mean based on the timestamp type?

ging
  • 219
  • 7
  • 20

1 Answers1

1

kafka_timestamp is for when you want to set a custom timestamp on an outbound record.

kafka_receivedTimestamp is populated from the incoming ConsumerRecord (which was set when the record was published).

A different header is used during header mapping to prevent inadvertent header propagation when sending a message which originated as an incoming message.

receive -> message -> processes -> send

If we used the same header, it would be possible for an application to set the same timestamp on the outbound message which most likely would not be incorrect. If that is actually what you want to do, then copy the received timestamp to the timestamp.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • So in short to summarize, kafka_receivedTimestamp is the publisher set time?What if publisher does not set any time? And Does it depend in kafka_timestampType at all? @Gary Russel – ging Dec 20 '22 at 13:55
  • I believe that if the publisher does not explicitly set a timestamp, the current time is set, using CREATE_TIME, or LOG_APPEND type, depending on the broker config https://kafka.apache.org/documentation/#brokerconfigs_log.message.timestamp.type (CREATE_TIME is the default). – Gary Russell Dec 20 '22 at 14:06