0

in order to debug our application, we save all incoming data (s3 sink) in a separate part of the graph (even before the timestamping/watermarking process). our data already includes timestamp (event timestamp), and before saving the data, we want to add one more field in which there will be a timestamp when the message actually got into the flink ( kind of processing time).

how best to do it? perhaps flink provides a special API for this now we are doing very simple new Date().getTime

Sergey Postument
  • 167
  • 12

1 Answers1

1

This is sometimes called ingestion time, btw. You are on your own to implement this; Flink doesn’t have anything built-in. What you’re doing seems fine.

David Anderson
  • 39,434
  • 4
  • 33
  • 60
  • looks like its available by using process function which have access to Context - https://stackoverflow.com/questions/49107932/apache-flink-how-to-get-timestamp-of-events-in-ingestion-time-mode – Sergey Postument Apr 08 '21 at 12:54
  • 1
    Support for ingestion-time timestamping was deprecated in Flink 1.10, which is why I didn't mention it. – David Anderson Apr 08 '21 at 14:12