We are running rabbit v3.8.3-1.el7, erlang v23.3.3.el7, kernel 3.10.0-1062.12.1.el7.x86_64, release Centos 7.7
I have three nodes that I would like in disc mode, cdvlhbqr23[1-3]
However I'm running into an issue after attempting to enable TLS on erlang.
[ cdvlhbqr231:rabbitmq ]
10.128.3.231 :: root -> rabbitmqctl status
Error: unable to perform an operation on node 'rabbit@cdvlhbqr231'. Please see diagnostics information and suggestions below.
Most common reasons for this are:
* Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
* CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
* Target node is not running
In addition to the diagnostics info below:
* See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
* Consult server logs on node rabbit@cdvlhbqr231
* If target node is configured to use long node names, don't forget to use --longnames with CLI tools
DIAGNOSTICS
===========
attempted to contact: [rabbit@cdvlhbqr231]
rabbit@cdvlhbqr231:
* connected to epmd (port 4369) on cdvlhbqr231
* epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic
* TCP connection succeeded but Erlang distribution failed
* suggestion: check if the Erlang cookie identical for all server nodes and CLI tools
* suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
* suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
* suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
Current node details:
* node name: 'rabbitmqcli-23412-rabbit@cdvlhbqr231'
* effective user's home directory: /var/lib/rabbitmq/
* Erlang cookie hash: MudCW7tn3FA5sTmC1FlR0g==
I've double checked the cookie file and it's identical across all nodes. All of the hostnames are correct and consistent across the nodes. So I assume this has to be directly the result of ssl / tls
Here's what the node config looks like:
[ cdvlhbqr231:rabbitmq ]
10.128.3.231 :: root -> cat /etc/rabbitmq/rabbitmq.config
[
{rabbit,
[
{vm_memory_high_watermark, 0.4},
{vm_memory_high_watermark_paging_ratio, 0.5},
{memory_alarms, true},
{disk_free_limit, 41686528},
{cluster_partition_handling, autoheal},
{tcp_listen_options,
[binary,
{packet, raw},
{reuseaddr, true},
{backlog, 128},
{nodelay, true},
{exit_on_close, false},
{keepalive, true}
]
},
{cluster_nodes, {['rabbit@cdvlhbqr231', 'rabbit@cdvlhbqr232', 'rabbit@cdvlhbqr233'], disc}},
{loopback_users, []},
{tcp_listeners, [{"0.0.0.0",5672}]},
{ssl_listeners, [{"0.0.0.0",5671}]},
{ssl_options, [
{cacertfile,"/etc/pki/tls/certs/ca-bundle.crt"},
{certfile,"/etc/rabbitmq/ssl/cert.pem"},
{keyfile,"/etc/rabbitmq/ssl/key.pem"},
{verify,verify_peer},
{versions, ['tlsv1.2']},
{fail_if_no_peer_cert,false}
]}
]
},
{rabbitmq_management,
[{
listener, [
{port, 15672},
{ip, "0.0.0.0"},
{ssl, true},
{ssl_opts, [
{cacertfile,"/etc/pki/tls/certs/ca-bundle.crt"},
{certfile,"/etc/rabbitmq/ssl/cert.pem"},
{keyfile,"/etc/rabbitmq/ssl/key.pem"},
{verify,verify_peer},
{versions, ['tlsv1.2']}
]}
]}
]
}
].
The private key is generated on the host and signed by an intermediate CA whose pub key is in the systems extracted cert bundle. We generate an "/etc/rabbitmq/ssl/allfile.pem" which is a bundle of the servers private key and signed cert.
The ssl environment for erlang is defined as the following:
[ cdvlhbqr231:rabbitmq ]
10.128.3.231 :: root -> cat rabbitmq-env.conf
# Obtaining of an Erlang ssl library path
export HOME=/var/lib/rabbitmq/
ERL_SSL_PATH=/usr/lib64/erlang/lib/ssl-9.6.2/ebin
# Add SSL-related environment vars for rabbitmq-server and rabbitmqctl
SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH \
-proto_dist inet_tls \
-ssl_dist_opt server_certfile '/etc/rabbitmq/ssl/allfile.pem' \
-ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"
# CLI
CTL_ERL_ARGS="-pa $ERL_SSL_PATH \
-proto_dist inet_tls \
-ssl_dist_opt server_certfile /etc/rabbitmq/ssl/allfile.pem \
-ssl_dist_opt server_secure_renegotiate true client_secure_renegotiate true"
I'm not entirely clear what's causing the issue. I thought I had followed the documentation to a T. Can anyone help me review this and see if there is anything obvious I'm missing, or any suggestions on how to trace down this problem?
