GlusterFS SSL not working

Question

I have been trying for hours to get GlusterFS SSL workong on the management and I/O path to no avail, so I hope you can help me.

I have tried it the simple way:

openssl genrsa -out glusterfs.key 4096
openssl req -new -x509 -key glusterfs.key -subj "/CN=Anyone" -out glusterfs.pem

And then I concatenated the glusterfs.pem from all peers to glusterfs.ca and distributed glusterfs.ca to all nodes. The following files exist:

/etc/ssl/glusterfs.key
/etc/ssl/glusterfs.pem
/etc/ssl/glusterfs.ca

I have created the file to enable SSL on the management path and restarted glusterd.

touch /var/lib/glusterd/secure-access
systemctl restart glusterd

My glusterd.log is now logging hundreds of entries (dozens per minute) showing an SSL error:

[2017-08-29 22:01:07.535725] E [socket.c:202:ssl_dump_error_stack] 0-socket.management:   error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
[2017-08-29 22:01:07.535760] E [socket.c:2519:socket_poller] 0-socket.management: server setup failed

The same error occurs in the volume log when I try to enable SSL on the I/O path and the same error also occurs when I try a more complex way of creating my glusterfs.* files. Example:

openssl genrsa 4096 > glusterfs.ca.key
openssl req -sha1 -new -x509 -nodes -days 3650 -key glusterfs.ca.key > glusterfs.ca
openssl req -sha1 -newkey rsa:4096 -days 1825 -nodes -keyout glusterfs.key > glusterfs-req.pem
openssl rsa -in glusterfs.key -out glusterfs.key
openssl x509 -sha1 -req -in glusterfs-req.pem -days 1825  -CA glusterfs.ca -CAkey glusterfs.ca.key -set_serial 01 > glusterfs.pem
openssl verify -CAfile glusterfs.ca glusterfs.pem

But this results in the same error as shown above, no matter what CN etc. I set and no matter whether I set the same or different CN on the peers.

I have also tried creating RSA keys with a key length of 2048 to no avail.

I am running out of ideas and I don't know what this error means in the context of GlusterFS:

SSL3_GET_RECORD:wrong version number

Disclaimer: I am running CentOS 7 64-bit and GlusterFS 3.11.3 (latest version as of this writing).

Answer 1

After spending a few hours trying to figure out what is wrong with the certificates, I found that, in my case, the problem was a stupid mistake: I haven't restarted glusterd on one of the two servers that were hosting the bricks. Both servers were therefore configured exactly the same, but one of them was unaware of the new SSL configuration.

If you encounter the same issue, the first thing is to run service glusterd restart on all the nodes. If it doesn't work, the other answer, as well a bug report may help.

Answer 2

I have fixed the problem this way (only for I/O path SSL support) ... the cipher list shown in the GlusterFS tutorial did not work it turns out and I have now set the following:

gluster volume set mailrep-volume ssl.cipher-list 'HIGH:!SSLv2:!SSLv3:!TLSv1:!TLSv1.1:TLSv1.2:!3DES:!RC4:!aNULL:!ADH'

Also I did not concate the .pem files from the servers but instead created my own self-signed CA as shown in the more complex example above. The .ca file created this way must be the same on all peers and when it is used to sign the glusterfs.pem files no concat certs are needed.

This is my complete volume config:

Volume Name: mailrep-volume
Type: Replicate
Volume ID: 907250fb-d83e-4fe7-a63b-636d952095ff
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eris:/mnt/glusterfs/vmail
Brick2: orcus:/mnt/glusterfs/vmail
Options Reconfigured:
ssl.cipher-list: HIGH:!SSLv2:!SSLv3:!TLSv1:!TLSv1.1:TLSv1.2:!3DES:!RC4:!aNULL:!ADH
nfs.disable: on
transport.address-family: inet
client.ssl: on
server.ssl: on
auth.ssl-allow: eris,orcus

To verify I can do ... (1) and see ... (2)

(1) cat /var/log/glusterfs/glustershd.log | grep SSL
(2) 0-mailrep-volume-client-1: SSL support on the I/O path is ENABLED
(2) 1-mailrep-volume-client-1: SSL support on the I/O path is ENABLED

The error from above (SSL3_GET_RECORD:wrong version number) is fixed now ONLY when using SSL on the I/O path. The problem still exists when I enable SSL on the management path but I don't know how to alter the cipher-lists for that to apply the same fix.

A little side note: The GlusterFS SSL tutorial seems to be horribly outdated. It is also stated that /etc/ssl/glusterfs.* are hardcoded file locations, but at least for SSL on the I/O path you can easily apply custom locations this way (tested and worked):

gluster volume set mailrep-volume ssl.private-key /path/to/your/certkey
gluster volume set mailrep-volume ssl.own-cert /path/to/your/certfile
gluster volume set mailrep-volume ssl.ca-list /path/to/your/cafile

I found those options in the Gluster source.

PS: I have also upgraded to GlusterFS 3.12.0 today (after the fix!)