0

Github Enterprise Server comes with a utility:

ghe-logs-tail

this tails all GHE Server logs simultaneously and prints the flow to the console for the user to view.

When trying to grep through these through, eg. for the "error" string like so:

ghe-logs-tail | grep --line-buffered -i "error"

The console will not print the flow and instead includes an error that some log files could not be opened for some reason.

$ ghe-logs-tail | grep --line-buffered -i "error"
/var/log/github/audit.log /var/log/github/auth.log /var/log/github/exceptions.log /var/log/github/gitauth.log /var/log/github/personal.log /var/log/github/production.log /var/log/github/resqued.log /var/log/github/unicorn.log /var/log/enterprise-manage/login_attempts.log /var/log/enterprise-manage/unicorn.log /var/log/github/auth.log /var/log/auth.log /var/log/nginx/alambic.assets.error.log /var/log/nginx/alambic.assets.log /var/log/nginx/alambic.avatars.error.log /var/log/nginx/alambic.avatars.log /var/log/nginx/alambic.error.log /var/log/nginx/alambic.log /var/log/nginx/avatars.error.log /var/log/nginx/avatars.log /var/log/nginx/credits.error.log /var/log/nginx/credits.log /var/log/nginx/enterprise-manage.error.log /var/log/nginx/enterprise-manage.log /var/log/nginx/error.log /var/log/nginx/gist.error.log /var/log/nginx/gist.log /var/log/nginx/github.error.log /var/log/nginx/github.log /var/log/nginx/pages.error.log /var/log/nginx/pages.log /var/log/nginx/raw.error.log /var/log/nginx/raw.log /var/log/nginx/render.error.log /var/log/nginx/render.log /var/log/nginx/static-maintenance.error.log /var/log/nginx/static-maintenance.log /var/log/nginx/storage.error.log /var/log/nginx/storage.log /data/user/common/ghe-config.log /var/log/syslog /var/log/dmesg /var/log/mysql/*.log /var/log/redis/redis.log /var/log/haproxy.log
==> /var/log/nginx/alambic.assets.error.log <==
==> /var/log/nginx/alambic.avatars.error.log <==
==> /var/log/nginx/alambic.error.log <==
==> /var/log/nginx/avatars.error.log <==
==> /var/log/nginx/credits.error.log <==
==> /var/log/nginx/enterprise-manage.error.log <==
==> /var/log/nginx/error.log <==
==> /var/log/nginx/gist.error.log <==
==> /var/log/nginx/github.error.log <==
tail: cannot open '/var/log/mysql/*.log' for reading==> /var/log/nginx/pages.error.log <==
==> /var/log/nginx/raw.error.log <==
: No such file or directory==> /var/log/nginx/render.error.log <==

==> /var/log/nginx/static-maintenance.error.log <==
==> /var/log/nginx/storage.error.log <==
tail: cannot open '/var/log/redis/redis.log' for reading: No such file or directory

What is the correct way to do this with a native tool like grep/awk?

I would obviously like to have all lines that contain the matching string printed out in real time to the console.

pythonInRelay
  • 99
  • 1
  • 8
  • Looks to me like it's doing exactly what you asked for, printing all lines that contain `error`, but some files that `ghe-logs-tail` is trying to tail don't exist or aren't readable so it's telling you about those (on stderr presumably). – Ed Morton Sep 08 '22 at 12:23

1 Answers1

0

you can filter out grep errors by routing them to the null device (blackhole)

... | grep ... 2>/dev/null
karakfa
  • 66,216
  • 7
  • 41
  • 56
  • Looks to me like the errors are coming from ghe-logs-tail, not grep. – Ed Morton Sep 08 '22 at 21:13
  • 1
    Oh, didn't pay attention. So should move it before the pipe then. If it's a stream consistent script it should work but have no idea what that is. – karakfa Sep 08 '22 at 22:30
  • Thanks for the reply. `ghe-logs-tail >2/dev/null | grep -i "error"` suppresses the errors but it doesn't fix the issue that some of the logs cannot be opened for some reason when using grep. – pythonInRelay Sep 09 '22 at 08:11
  • @pythonInRelay - again, it's not **grep** that can't open any files, read the error messages "**tail**: cannot open..." plus the fact that adding `>2/dev/null` after the tail command suppresses them. Those errors will be present whether you pipe to grep or not. – Ed Morton Sep 09 '22 at 12:17
  • @EdMorton how then do I avoid these errors? I don't care about surpressing them. I want still to be able to open these log files and read them in real-time using grep. – pythonInRelay Sep 12 '22 at 11:11
  • you have to fix `ghe-logs-tail` in order to not to get errors. – karakfa Sep 12 '22 at 13:21