12

Timestamp format in logs

Most log lines contain a timestamp and event description:

[When] [What]

e.g.:

[23/Jul/2013:19:35:11 +0000] Processing started.
[23/Jul/2013:19:36:11 +0000] Processed 1000 items.
[23/Jul/2013:19:37:11 +0000] Processing finished successfully.

I am trying to find a standard timestamp for my log lines. My criteria is:

  1. Human readable. I want to easily understand when did the event happen.
  2. Alphabetically sortable. When I grep events from a few files and sort them using POSIX sort or even word/excel, I want the alphabetical sort to adhere to chronological sort. For example, [23/Jul/2012:19:35:11 +0000] and [22/Jul/2013:19:35:11 +0000] are not sortable - the 2013 line would appear before the 2012 line.
  3. Easily parsable by all common languages. The timestamp should be easily parsed using standard strptime if the log is processed by a script.

The only standard I've found so far is ISO_8601, which has many variants (e.g. 2007-04-05T14:30Z and 2007-03-01T13:00:00Z), and lacks a definite standard for log line events.

Could you recommend a standard timestamp format for log lines?

Adam Matan
  • 128,757
  • 147
  • 397
  • 562
  • 2
    [rfc 3339 defines a profile of ISO 8601](http://tools.ietf.org/html/rfc3339#section-5) that comes close to 1-3 – jfs Jul 23 '13 at 20:47
  • Thanks. It recommends something like `1985-04-12T23:20:50.52Z` or `1996-12-19T16:39:57-08:00`. The only thing I don't like is the `T` in the middle; the separation between date and time isn't clear enough. Perhaps I should use a space. – Adam Matan Jul 23 '13 at 23:04
  • 1
    The rfc mentions that applications may use space instead of "T" for readability – jfs Jul 24 '13 at 07:12
  • Great, so I'll go for that. If you post your comment as an answer I'll accept it – Adam Matan Jul 24 '13 at 18:55

2 Answers2

20

enter image description here

@J.F. Sebestian - Thanks for your comment.

After some research I chose RFC 3339 / ISO 8601 in UTC, e.g.:

date -u "+[%Y-%m-%d %H:%M:%S%z (%Z)]"       # Space separated with tz abbreviation
[2013-07-31 23:56:34+0000 (UTC)]                   

date -u "+[%Y-%m-%d %H:%M:%S.%N %z (%Z)]"  # Space separated with nanoseconds and tz abbreviation
[2013-07-31 23:56:34.812572000 +0000 (UTC)]

Features:

  • Sortable (Most significant date item is on the left)
  • Readable
  • Unambiguous, time zone clearly stated
  • Delimited by [,], useful for regexing the date away
  • Easily parsable
  • Accurate: uses nanoseconds (might be practically milliseconds on some machines, which is good enough)

I've also created a nice github project that helps with date formatting - feel free to take a look and suggest your own formats!

Adam Matan
  • 128,757
  • 147
  • 397
  • 562
2

The unix date command has such an option. Use

date -Iseconds

or

date -Ins

The manpage says:

   -I[FMT], --iso-8601[=FMT]
          output  date/time  in  ISO  8601  format.   FMT='date' for date only (the default),
          'hours', 'minutes', 'seconds', or 'ns' for date and time to  the  indicated  preci‐
          sion.  Example: 2006-08-14T02:34:56-0600
Black
  • 343
  • 1
  • 4
  • 12