Logstash-forwarder is throwing SSL errors

Question

I got this task handed over to by my colleage and this is the background.

He got ELK (Elasticsearch, Logstash and Kibana) stack working with our RHEL 6.2 servers, by using the regular method of configuring the Logstash on the server and Logstash-forwarder on all the agents. Everything is working fine as expected, except for two logstash agent nodes. It is not indexing data from these two nodes and these two agent nodes are slightly different from the other agents in the environment. All the agents (that are working) are running RHEL 6.2 machine with 'OpenSSL 1.0.0-fips', while the two troublesome agents are running RHEL 5.5 with 'OpenSSL 0.9.8e-fips-rhel5'. Now, let me explain the issues that these two nodes were facing when I started working on this.

When we restart the logstash forwarder service with '/etc/init.d/logstash-forwarder restart', it throws the following error:

2014/10/01 09:21:23.898917 Failed to tls handshake with 10.x.x.x x509: cannot validate certificate for 10.x.x.x because it doesn't contain any IP SANs 2014/10/01 09:21:24.900502 Connecting to [10.x.x.x]:5000 (10.x.x.x) 2014/10/01 09:21:24.902081 Failed to tls handshake with 10.x.x.x x509: cannot validate certificate for 10.x.x.x because it doesn't contain any IP SANs 2014/10/01 09:21:25.903970 Connecting to [10.x.x.x]:5000 (10.x.x.x) 2014/10/01 09:21:25.905708 Failed to tls handshake with 10.x.x.x x509: cannot validate certificate for 10.x.x.x because it doesn't contain any IP SANs

I understand that this is because the agent is trying to connect to the server through SSL using its IP address, so I recreated the certificates again with IP SANs and then it started throwing me following errors, not only with the 5.2 machines, but also with 6.2 machines:

2014/10/03 17:09:12.403253 Failed to tls handshake with 10.x.x.x x509: certificate signed by unknown authority 2014/10/03 17:09:13.404974 Connecting to [10.x.x.x]:5000 (10.x.x.x) 2014/10/03 17:09:13.428156 Failed to tls handshake with 10.x.x.x x509: certificate signed by unknown authority 2014/10/03 17:09:14.429648 Connecting to [10.x.x.x]:5000 (10.x.x.x) 2014/10/03 17:09:14.442006 Failed to tls handshake with 10.x.x.x x509: certificate signed by unknown authority

I tried to add the certificate to the ca-bundle.crt by appending the new logstash certificate to the CA file with:

'openssl x509 -in certs/logstash-forwarder.crt -text >> certs/ca-bundle.crt'.

I have a feeling that I will be able to fix the whole issue if I am able to make the 'IP SAN'd certificates add to the trusted certificates list of all the agent nodes. My questions are:

One more thing, the Logstash server is running RHEL 6.2. And all the agent nodes (working and non-working) have the same version of logstash-forwarder installed - logstash-forwarder-0.3.1-1.x86_64.rpm. What is the correct method to make these agent nodes accept the IP SANs certificates? Am I understanding issue correctly? Is it some incompatibility between the RHEL 6.x and RHEL 5.x machines?

I have the following entries in the /etc/logstash-forwarder

[root@name2 ~]# cat /etc/logstash-forwarder { "network": { "servers": [ "10.x.x.x:5000" ], "timeout": 15, "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt", "ssl strict verify": "false" }, "files": [ { "paths": [ "/var/log/maillog" ], "fields": { "type": "syslog" } } ] } [root@name2 ~]#

Thanks in advance.

Answer 1

The certificate you are using does not have any valid IP SAN's as mentioned in the message:

Failed to tls handshake with x.x.x.x x509: cannot validate certificate for x.x.x.x because 
it doesn't contain any IP SANs  

If you connect using an IP address then your certificate must contain a matching IP SAN to pass validation with Go 1.3 and higher. This is not (yet?) mentioned in any README file or documentation but there are some issues ( #226 , #221 ) open at the project github repo.
To permit IP address as the server name, the SSL cert must include IP address as a subjectAltName field.

To solve that you can use following procedure for the creation of the SSL cert and key:

1. Create a file notsec.cfr (or any other name) containing output like:

   [req]
   distinguished_name = req_distinguished_name
   x509_extensions = v3_req
   prompt = no
   
   [req_distinguished_name]
   C = TG
   ST = Togo
   L =  Lome
   O = Private company
   CN = *
   
   [v3_req]
   subjectKeyIdentifier = hash
   authorityKeyIdentifier = keyid,issuer
   basicConstraints = CA:TRUE
   subjectAltName = @alt_names
   
   [alt_names]
   DNS.1 = *
   DNS.2 = *.*
   DNS.3 = *.*.*
   DNS.4 = *.*.*.*
   DNS.5 = *.*.*.*.*
   DNS.6 = *.*.*.*.*.*
   DNS.7 = *.*.*.*.*.*.*
   IP.1 = 192.168.1.1
   IP.2 = 192.168.69.14  
   

If you connect via host names, you can remove the IP SAN's, otherwise add your logstash server IP address.

2. Create the certificate and key with following command (using the file from point 1):

   openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout notsecure.key -out notsecure.crt -config notsec.cnf -days 1825
   

This will create a kind of wildcard certificate accepting any hostname and the IP addresses mentioned it that file. Of course this is just a simple example and you wil need to adjust the settings to your needs.