Vertica database creation fails because node is down

Question

I am trying to create database in fresh 3 node cluster, but got following error

*** Creating database: test ***
        172.16.10.47 OK [vertica][(6, 0, 1)][007][x86_64]
        172.16.10.76 OK [vertica][(6, 0, 1)][007][x86_64]
        172.16.10.55 OK [vertica][(6, 0, 1)][007][x86_64]
    Checking full connectivity
Creating database test
Node Status: v_test_node0001: (DOWN)
Node Status: v_test_node0001: (INITIALIZING)
Node Status: v_test_node0001: (VALIDATING LICENSE)
Node Status: v_test_node0001: (UP)
Creating database nodes
Creating node v_test_node0002 (host 172.16.10.76)
Creating node v_test_node0003 (host 172.16.10.55)
Node Status: v_test_node0001: (UP) v_test_node0002: (DOWN) v_test_node0003: (DOWN)
Node Status: v_test_node0001: (UP) v_test_node0002: (DOWN) v_test_node0003: (DOWN)
Node Status: v_test_node0001: (UP) v_test_node0002: (DOWN) v_test_node0003: (DOWN)
Node Status: v_test_node0001: (UP) v_test_node0002: (DOWN) v_test_node0003: (DOWN)
Node Status: v_test_node0001: (UP) v_test_node0002: (DOWN) v_test_node0003: (DOWN)
Node Status: v_test_node0001: (UP) v_test_node0002: (DOWN) v_test_node0003: (DOWN)
Node Status: v_test_node0001: (UP) v_test_node0002: (DOWN) v_test_node0003: (DOWN)
Node Status: v_test_node0001: (UP) v_test_node0002: (DOWN) v_test_node0003: (DOWN)
Node Status: v_test_node0001: (UP) v_test_node0002: (DOWN) v_test_node0003: (DOWN)
Node Status: v_test_node0001: (UP) v_test_node0002: (DOWN) v_test_node0003: (DOWN)
Nodes UP: v_test_node0001
Nodes DOWN: v_test_node0002, v_test_node0003 (may be still initializing).

I did try stoping and starting the vertica_agent on all 3 nodes, but it didn't help.

also service status return not OK

[dbadmin@vertica001 ~]$ sudo service verticad status
Vertica: status not OK

iptables stopped and Selinux disabled

Anyone know what wrong here ?

Seems like node 1 doesn't have access to node 2 and 3 i.e. your firewall closed [Vertica ports](https://my.vertica.com/docs/7.1.x/HTML/Content/Authoring/InstallationGuide/BeforeYouInstall/EnsurePortsAreAvailable.htm) on all nodes. So, check it. — Maxim, Feb 19 '16 at 19:31
I have no firewall rules for these machines. I am able to telnet 5433, 4803, but not 5434, 5450 & 6543. — roy, Feb 19 '16 at 19:46
I dont have any rules set up. Nothing is listening on 5434, 5450 & 6543. — roy, Feb 19 '16 at 20:17

score 0 · Answer 1 · answered Feb 21 '16 at 23:24

0

You should start by looking at your logs :

/opt/vertica/logs/install.log
/opt/vertica/logs/adminTools.log
/opt/vertica/logs/adminTools.errors

For here you will find what is going bad !

answered Feb 21 '16 at 23:24

Up_One

5,213
3
33
65

score 0 · Answer 2 · answered Feb 22 '16 at 13:12

0

+1 to Up_One's response.

I will be surprised though if this doesn't turn out to be port communication issue (firewall or otherwise). If this AWS or bare metal?

Even if iptable -L output returns an empty table, I would recommend stopping iptables.

Also, is spread keeping up on these two nodes?

$ ps -ef | grep spread

or

$ /etc/init.d/spreadd status

Thanks

answered Feb 22 '16 at 13:12

Pravesh

11
1

I have iptables stopped on all three servers. SElinux disabled. spreadd is running on all nodes – roy Feb 22 '16 at 17:00
Anything suggestive in admintools-.log? It would also be interesting to see if startup goes to a point that it writes something to Vertica.log on nodes 2 & 3. – Pravesh Feb 22 '16 at 17:12

Vertica database creation fails because node is down

2 Answers2