Pipe stdout to loggly via simple bash script

Question

I want to log stdout from some programs to loggly. There are a few simple utilities out there that I've found (e.g. https://github.com/meatballhat/loggly-pipe and https://github.com/segmentio/loggly-cat), but they seem like they're overkill.

Could I get away with doing something this simple:

log.sh:

#!/bin/bash
while read line
do
    echo "$line"
    curl -H "content-type:text/plain" -d "$line" https://logs-01.loggly.com/inputs/<my-token>/tag/tag1,tag2/ >/dev/null 2>&1
done < /dev/stdin

Then I run my program and pipe it to my loggly logging script:

./my_script.sh | ./log.sh

This seems to work okay, but I wonder if all the complicated-ness of the other solutions out there is necessary for some reason?

Could anything go wrong here?

Thanks!

The `< /dev/stdin` part is unnecessary, by the way. – John Zwinck Jan 09 '15 at 07:40 — John Zwinck, Jan 09 '15 at 07:40
Excellent. Less code more better. – Nate Jan 09 '15 at 07:43 — Nate, Jan 09 '15 at 07:43

score 1 · Answer 1 · answered Jan 09 '15 at 07:43

1

Think about what your script does. It runs curl once per line of input.

Think about what that means.

If you log 10K lines, you'll then spawn 10K processes. Which will initiate 10K TCP connections. This is a massive waste of computing resources.

Also, you don't handle errors at all (in fact, you actively hide them by sending curl's stderr to /dev/null!). This means the script is not only inefficient, it is unreliable.

I recommend starting all Bash scripts with set -eu to exit on unhandled errors, but that's just one step, not a complete fix for the above.

answered Jan 09 '15 at 07:43

John Zwinck

239,568
38
324
436

So, perhaps batching TCP connections would be less of a waste, and I assume the other solutions do that to some extent. I don't necessarily want to handle errors, I just want it to not break or barf if it's not working for any reason. I'd rather not get logs than break something else. If I exit with set -eu, would the first my_script.sh continue running? – Nate Jan 10 '15 at 01:43
No, if the second script in the pipeline stops, the first one will receive SIGPIPE which will in all likelihood stop it also. It might be a good idea to write the failed log lines to a file or something. – John Zwinck Jan 10 '15 at 02:05

Pipe stdout to loggly via simple bash script

1 Answers1