0

I am sending data to 9516 port on nginx server . From this server I want to forward it to two syslog-ng servers using roundrobin method. I have below nginx.conf file, with this I am able to load balance my kibana webpage but I am not sure where and how I can read 9516 port and then forward it to syslog-ng machine .

user www-data;
worker_processes 4;
pid /run/nginx.pid;

events {
        worker_connections 768;
}

http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
        gzip on;
        gzip_disable "msie6";
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
upstream kibana {
    server1:30001;
    server2:30001;
  }

  server { # simple load balancing
    listen          80;
    server_name     nginxIP;
   # SSL Certificate, Key and Settings
        ssl_certificate /etc/pki/tls/certs/ELK-Stack.crt ;
        ssl_certificate_key /etc/pki/tls/private/ELK-Stack.key;
        ssl_session_cache shared:SSL:10m;

        # Basic authentication using the account created with htpasswd
        auth_basic "Restricted Access";
        auth_basic_user_file /etc/nginx/htpasswd.users;
    location / {
      proxy_pass      http://kibana;
    }
}

I took help from above and tried to add below but I am getting an error 

upstream syslog-ng {
    syslog-ng-server1:9516;
    syslog-ng-server2:9516;
  }

  server { # simple load balancing
    listen          9516;
    server_name     nginxIP;
    location / {
      proxy_pass      http://syslog-ng;
    }
}

Error While sending log on 9516 port using netcat command

<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>

Please help.

Regards VG

user3332404
  • 411
  • 1
  • 4
  • 7
  • Any suggestion from anyone please ? – user3332404 Jun 12 '17 at 10:19
  • So , to summarize I want to send logs on 9516 port to Nginx server which will further forward data to two syslog-ng machines using round robin method . – user3332404 Jun 12 '17 at 10:22
  • in more simple way this is what I want to achieve USER -> POST DATA -> NGINX INSTANCE ----REDIRECT(load balance) ---> syslog-SERVER 1 AND syslog-SERVER 2 – user3332404 Jun 12 '17 at 12:25
  • Hi, I'm not sure I get your usecase. You want to send data from nginx to syslog-ng in load-balance fashion? Why not simply have syslog-ng read the logs of nginx? Why do you need loadbalancing, is the message rate so high? – Robert Fekete Jun 13 '17 at 11:37
  • Hi Robert, sorry if I was not clear with my requirement . Actually I am getting logs from archsight SIEM solution on 9516 port so it is like 100 GB a day and most of the logs are of firewall logs . Now since I want to segregate the logs based on there values I am using syslog-ng so firewall logs stored in filrewall directory and windows logs into windows and so on . So my plan to land these logs on nginx server on 9516 port and then from nginx loadbalance these huge volume to 2 syslo-ng servers on 9516.This will also help me incase fail-over if one channel is down load will shift to another. – user3332404 Jun 14 '17 at 02:39
  • Hi, Are you sure that you need nginx? As I understand, you are trying to send logs from nginx to syslog-ng using HTTP - this is currently not possible, syslog-ng does not have an HTTP source. If I understand your needs correctly, you want to get the logs from Arcsight and send them to Elasticsearch/Kibana. You could forward the logs from Arcsight (using a syslog protocol) to syslog-ng, which can filter the messages and send them directly to Elasticsearch. Version 3.10 of syslog-ng Open Source Edition (to be released in a few weeks) will support load-balancing to Elasticsearch. – Robert Fekete Jun 14 '17 at 07:15
  • So why I need Nginx , first I need LDAP authentication to kiabana, second I need reverse proxy , third I need high availability of the full ELK stack . So I have 8 total machines 2-ES, 2-LS ,2 for kibana and 2 Nginx so, via nginx I am achieving high availability. Elasticserch is fine as it is auto loadbalanced but Logstash and Kibana are not . Now why I cannot send logs directly to elasticsearch as I am doing log parsing using logstash and on the same logstash machines I have syslog-ng which puts data in different directories . With HA proxy I achieved it but in Nginx I am not able to . – user3332404 Jun 14 '17 at 07:48

0 Answers0