12

Having a long running program that continuously writes to a logfile - how is it possible, disregarding any buffering issues, to add a date string to each line written to that file using a linux script?

I would imagine something like this:

tail -f logfile | ADD_DATE_TO_EACH_LINE > logfile2

The input would be something like that:

abc
def
ghi
jkl

The output should be similar to that:

2011-06-16 18:30:59 abc
2011-06-16 18:31:00 def
2011-06-16 18:35:21 ghi
2011-06-16 18:40:15 jkl
bmk
  • 13,849
  • 5
  • 37
  • 46

8 Answers8

27

With perl:

command 2>&1 | perl -pe 'print scalar(localtime()), " ";'

With gawk:

command 2>&1 | awk '{ print strftime(), $0; fflush() }'

Replace command with tail -f logfile for your specific example. Or, perhaps you could just redirect the original program's stdout/stderr to the above pipe.

Steve Prentice
  • 23,230
  • 11
  • 54
  • 55
  • The benefit you get with this answer is that it doesn't spawn a new date process with each line, but both work well. – Steve Prentice Jun 16 '11 at 17:32
  • `perl -pne` ?? You don't want `-n` as well as `-p`. – glenn jackman Jun 16 '11 at 18:39
  • Thanks! I don't really know perl. I just cut and paste other crud from the internet. :) Last time I used perl for real was over a decade ago! I usually use the gawk provided answer above, but included the perl one too in case the OP was not on linux. – Steve Prentice Jun 16 '11 at 20:54
  • @StevePrentice When I do the first command, it hangs. I'm running `tail -f test-text.rtf 2>&1 | perl -pe 'print scalar(localtime()), " ";' > test-text.rtf` Any idea why? – RNdev Nov 18 '19 at 19:00
8

Try

tail -f logfile | while read line; do echo `date` "$line" ; done
aioobe
  • 413,195
  • 112
  • 811
  • 826
  • 3
    +1 Just one note though: using `date +"%y-%m-%d %H:%M:%S"` instead of simply `date` would result in the exact output he expects; and you could also add the redirect to the output file in the end. (It's actually two notes) :) – Costi Ciudatu Jun 16 '11 at 17:09
  • Thank you, that works. And if you add `"` signs to `$line` the shell will not expand `*` signs to the directory content ;) – bmk Jun 16 '11 at 17:14
  • 1
    the problem with this solution is : when the input buffer given to the 'while read line' loop is too big, some lines will be truncate. – Zskdan Oct 18 '13 at 12:15
7

You can try this

cat /etc/motd | xargs -d"\n" -I {} date +"%Y-%m-%d %H:%M:%S {}"

Example output:

2013-02-26 15:13:57 
2013-02-26 15:13:57 The programs included with the Debian GNU/Linux system are free software;
2013-02-26 15:13:57 the exact distribution terms for each program are described in the
2013-02-26 15:13:57 individual files in /usr/share/doc/*/copyright.
2013-02-26 15:13:57 
2013-02-26 15:13:57 Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
2013-02-26 15:13:57 permitted by applicable law.
Felipe
  • 161
  • 2
  • 7
  • @Vouze You can fix that with `sed` in between, or actually rather instead of the [useless use of `cat`](http://www.iki.fi/era/unix/award.html); `sed 's/%/%%/g' /etc/motd | xargs ...` – tripleee Aug 31 '16 at 04:58
5

A tool exists for that exact purpose, it’s ts (see man ts)

For example, with your logfile:

tail -f logfile | ts '%Y-%m-%d %H:%M:%S'

Also works with any software writing on the standard output, of course:

./blabla | ts '%Y-%m-%d %H:%M:%S'

If needed, you can add sub-second precision, with %.S instead of %S

user13042276
  • 51
  • 1
  • 1
  • `ts` is in the package `moreutils` on Ubuntu, and similar, installed via `sudo apt install moreutils`. – pbhj Mar 28 '21 at 00:59
3

A little lengthy, but here is what I came up with:

tail -f logfile | sed -u 's/%/%%/g' | xargs -I {} date +"%Y-%m-%d %H:%M:%S {}"
Andrew Clark
  • 202,379
  • 35
  • 273
  • 306
  • That's a very good solution. But what's the `sed -u 's/%/%%/g'` for? Do the `%` signs have a special meaning for `xargs`? – bmk Jun 16 '11 at 17:30
  • @bmk This method works by putting the entire line into the format string for `date`, where `%` has a special meaning. For example if the line in the log contained `%Y` then it would get replaced by the year, but changing it to `%%Y` with `sed` will correctly output `%Y`. – Andrew Clark Jun 16 '11 at 17:35
  • 1
    I have one additional remark: If there are single quotes (`'`) in the input stream `xargs` terminates with following error message: `xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option` therefore I changed the command to `tail -f logfile | sed -u 's/%/%%/g' | tr "\n" "\0" | xargs -0 -I {} date +"%Y-%m-%d %H:%M:%S {}"`. – bmk Jun 20 '11 at 12:31
0

Or you can use python ...

cat /dev/urandom | python -c "from __future__ import print_function; import sys; import datetime; map(lambda x: print(datetime.datetime.now(), x), [line for line in sys.stdin.readlines()])"

Or use gnu screen 

screen -a ls -lh -L -U -X command

First you need to enable logging and timestamp on your ~/.screenrc.

logfile /tmp/screen-%S-%n.log
logtstamp on
Jorge Niedbalski R.
  • 604
  • 4
  • 12
0

Can you configure the long running program to write it's output to the standard output and not to the logfile? In this case it would be easy to pipe the output to a script that first writes the current timestamp and then the entry.

If that is impossible, it may help to periodically (e.g. every second) read the logfile content, copy each line to another file (adding the current timestamp) and then deleting the logfile. This may, however, impose losing logfile entries that are written between reading and deleting the file :(

BurninLeo
  • 4,240
  • 4
  • 39
  • 56
  • I cannot configure the program to log to standard output because it is (resp. can be used as) an interactive program. But actually executing `tail -f` is in my opinion more or less the same as directly writing to stdout. Actually I'm looking for that "script that first writes the current timestamp and then the entry". – bmk Jun 16 '11 at 17:09
  • you could use a FIFO then. The Program will write everything to the named pipe (fifo) and you can read from it the same time, append the date and write to the real logfile. – reox Sep 29 '14 at 12:50
0

I've seen elsewhere a suggestion, if timeliness is important, that one should avoid calling other programs from BASH and use a built-in:

printf '%(%F %T)T\n'

%T is the same format strings from date (see man date), AFAICT. Example:

output="Err, Miss Crabapple, I just ate my worm!"; \
printf '\n%(%F %s)T'; printf "%s \n" " $output"

will output:

2021-03-28 1616894012 Err, Miss Crabapple, I just ate my worm!

I've just noticed a comment on that link has a suggestion to use gawk for timestamp top speeds.

pbhj
  • 276
  • 3
  • 15