0

I want to push logs, produced by nginx to influxdb. To handle this, I have an agent, which can format it, extract some geodata, produce geohash, etc..

The problem is that nginx and agent are running in separate docker containers, and nginx is a producer, that is only capable of writing to filesystem, but agent is only capable of reading from network (eg some tcp stream, websocket or http, as it is usually done in webdev)

I dont want to share access.log volume between containers, wasting space, and long-parsing kilometers of logs..

How to make nginx write log directly to some tcp socket?

xakepp35
  • 2,878
  • 7
  • 26
  • 54

1 Answers1

3

nginx write log directly to some tcp socket is not safe ,
as socket writing may slowdown or even block nginx

nginx can be configured to logging to syslog
if the agent is capable of reading syslog 

access_log syslog:server=syslog_server_ip:sys_log_server_port,facility=local7,tag=nginx,severity=info;

if you are determined to do custom forwarding , here is another simple but buggy way
configure nginx write to a named pipe, send it out with ncat 

mkfifo /var/log/nginx/access.log  
ncat  agent_ip agent_port 0</var/log/nginx/access.log

this is buggy because ncat must be manually restarted when disconnection occurs
supervisor can help on restarting
but a few log lines may lost when they are hold by ncat but do not send out over network

flume is a better tool to do such log forwarding

James Li
  • 469
  • 3
  • 7
  • Thanks. I've finished up implementing syslog server, which parses nginx output to influxdb format, adds geohash and sends out to influxdb for mapping. And sometimes writing to HDD /var/log/nginx may slow up, on contrast with "writing to locally-deployed app"... – xakepp35 Sep 04 '19 at 14:36
  • @xakepp35 Well, you could have simply used [fluentd](https://www.fluentd.org) – Markus W Mahlberg Oct 03 '19 at 09:21