3

Trying to get OpsCenter 5.2.2 after fresh install to pick up data from two new clusters. Both cluster are freshly installed and runs fine:

[root@d1r1n1 ~]# metatool status
Note: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: OPA3
================
Status=Up/Down  
|/ State=Normal/Leaving/Joining/Moving  
--  Address      Load       Tokens  Owns   Host ID                               Rack
UN  10.45.68.13  136.01 KB  1       7.1%   692e6062-388d-49f4-93de-380405e1495f  R3.3.16
UN  10.45.68.12  150.64 KB  1       7.1%   20534507-1924-4240-a980-d4296566d48d  R3.3.16
UN  10.45.68.14  136.34 KB  1       7.1%   6bcdf08f-f49e-438e-b102-ac1b132cd3af  R3.3.16
UN  10.45.68.9   155.83 KB  1       7.1%   5984016d-7ab4-4cfc-99a7-1fb44e2525d5  R3.3.16
UN  10.45.68.8   140.67 KB  1       7.1%   bea12c79-5856-4dd6-8c2e-e2498229ab1e  R3.3.16
UN  10.45.68.11  150.44 KB  1       7.1%   58082d4c-c5bb-42b2-abfc-472602e6fd3c  R3.3.16
UN  10.45.68.10  155.61 KB  1       7.1%   489c3a11-2b7e-4dc0-b023-fca1b611d7d5  R3.3.16
UN  10.45.68.5   134.14 KB  1       7.1%   b42e3bb5-d033-4cd2-9a4d-d68e12026403  R3.3.16
UN  10.45.68.4   136.04 KB  1       7.1%   ad33485c-e4f8-40b8-804f-363ea326319c  R3.3.16
UN  10.45.68.7   150.67 KB  1       7.1%   174040b9-ea42-43c2-88f2-26c18ebeef15  R3.3.16
UN  10.45.68.6   134.33 KB  1       7.1%   7da26426-a4d6-4d21-b1e9-ac2a5896a453  R3.3.16
UN  10.45.68.1   141.33 KB  1       7.1%   466832c8-95b7-4cd9-8ace-87624478f79a  R3.3.16
UN  10.45.68.3   126.15 KB  1       7.1%   99a2332d-97d2-46e8-b93d-1986e7161789  R3.3.16
UN  10.45.68.2   126.69 KB  1       7.1%   03c62604-9a6f-42be-8d77-0a8f4113a307  R3.3.16

[root@d1r1n1 ~]# blobtool status
Note: Ownership information does not include topology; for complete information, specify a keyspace
Datacenter: OPA3
================
Status=Up/Down  
|/ State=Normal/Leaving/Joining/Moving  
--  Address      Load       Tokens  Owns   Host ID                               Rack
UN  10.45.68.13  104.56 KB  1       7.1%   103038fc-dce8-455b-84c4-5d1abee2a9d2  R3.3.16
UN  10.45.68.12  105 KB     1       7.1%   fb24a3ff-69fc-47d6-b434-db6e00aaea98  R3.3.16
UN  10.45.68.14  105.05 KB  1       7.1%   776d8a1b-9f63-4a04-93fc-1cb8b3439997  R3.3.16
UN  10.45.68.9   121.56 KB  1       7.1%   3a8d9139-67af-4c14-bfb8-880517323e5b  R3.3.16
UN  10.45.68.8   106.51 KB  1       7.1%   c41e9965-2a27-45cb-aa3e-d9f9b59cbc98  R3.3.16
UN  10.45.68.11  119.31 KB  1       7.1%   7528c8f6-aa79-4776-8a98-5c2890f2cf3d  R3.3.16
UN  10.45.68.10  119.29 KB  1       7.1%   12109e0b-e25e-45f1-b891-fe6debe91891  R3.3.16
UN  10.45.68.5   105.67 KB  1       7.1%   378a6f68-9014-4f04-9004-2cc1a0b607a9  R3.3.16
UN  10.45.68.4   117.05 KB  1       7.1%   7128ae20-186e-4b63-a284-4949e6596c11  R3.3.16
UN  10.45.68.7   106.6 KB   1       7.1%   165ade74-c5b2-4086-961c-8fe14a266379  R3.3.16
UN  10.45.68.6   106.63 KB  1       7.1%   744a6c72-b0fb-4db7-8582-a82a62ad74e5  R3.3.16
UN  10.45.68.1   113.92 KB  1       7.1%   3c4805f1-e606-450f-9c98-68b1ea7f9d4f  R3.3.16
UN  10.45.68.3   116.47 KB  1       7.1%   6b0dc9fa-576f-4832-b16e-0051b0d6d48a  R3.3.16
UN  10.45.68.2   101.05 KB  1       7.1%   2eb608f3-9499-4862-be0c-5154115ab4ea  R3.3.16

[root@d1r1n1 ~]# alias | grep tool
alias blobtool='nodetool -h localhost -p 8081'
alias metatool='nodetool -h localhost -p 8082'

Defined my two clusters in OpsCenter only whenever data are send from the agents; I get this annoying warnings in /var/log/opscenter/opscenter.log:

2015-10-30 18:09:58+0100 []  WARN: Unable to find a matching cluster for node with IP d1r1n1; the message was {u'os-load': 0.040000000000000001}. This usually indicates that an OpsCenter agent is still running on an old node that was decommissioned or is part of a cluster that OpsCenter is no longer monitoring.

Wonder if the host name d1r1n1 instead of it's IP is an issue though:

[root@opsc opscenter]# dig +short d1r1n1
[root@opsc opscenter]# dig +short +search d1r1n1
10.45.68.1

Not sure why and how to fix this, hints are appreciated.

TIA!

Screen Dump of Overview Dashboard here!

Steffen Winther Sørensen
  • 343
  • 1
  • 11
  • 1
    Okay I found that I needed to use IP and not hostname as initial in my agents address.yaml then agents will be connected to opscenter – Steffen Winther Sørensen Oct 31 '15 at 10:21
  • 1
    Also I moved one cluster to another IP stack on all nodes thus more properly have each node multihoming on separate IP stacks, off course I also added new secondary IP stack on my opscenter box. This way I have two separated 'vlans' with each a cassandra cluster ring and an agents nicely connected and showing up on opscenter. [all_agents_connected](https://dl.dropboxusercontent.com/u/13225502/opscenter_multihomed_agents_connected.tiff) – Steffen Winther Sørensen Oct 31 '15 at 10:38
  • 1
    Though agents are connected from my two rings across the same 14 physical nodes [cass_clusters](https://dl.dropboxusercontent.com/u/13225502/cassandra_multiple_rings.tiff), they get weird 'names' in opscenter, my MetaCl cluster with 10.45.68/24 IPs shows with these names [MetaCl_weird_names](https://dl.dropboxusercontent.com/u/13225502/opscenter_multihomed_weird_names_MetaCl.tiff) and my BlobCl cluster with 10.3.2/24 IPs shows with these names [BlobCl_weird_names](https://dl.dropboxusercontent.com/u/13225502/opscenter_multihomed_weird_names_BlobCl.tiff). How to fix this? Hints anyone? – Steffen Winther Sørensen Oct 31 '15 at 10:45
  • ;) damn me self, had a typo in agent api_port on the cluster config, correcting this I now can run actions against agent stomp api (compact etc.) and getting the same names (d1r1nX.nat.tdcfoo), though weirdly not PTR dns names for the IP addresses involved (d1r1nX.meta or d1r1nX.blob) as expected. How are OpsCenter trying to map IPs to figure a name of a node? – Steffen Winther Sørensen Oct 31 '15 at 13:30
  • (sorry do know how to format code blocks in comments ;) Got 3x vlans on OpsCenter: ` [root@opsc ~]# hostname -I 10.45.68.252 10.45.70.8 10.3.2.252 [root@opsc ~]# hostname -A opsc.meta opsc.nat.tdcfoo ops.blob [root@opsc ~]# netstat -lnt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 10.45.70.8:8080 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:61619 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:61620 0.0.0.0:* LISTEN` How does OpsCenter pick names for agent IP addresses if not through gethostbyaddress – Steffen Winther Sørensen Oct 31 '15 at 13:49
  • Regarding formatting, see this link which should help: http://stackoverflow.com/help/formatting – markc Nov 11 '15 at 21:24
  • Just a thought but have you tried entering two unique hostnames for the two unique IPs on the agent host machine? – markc Nov 11 '15 at 21:28

0 Answers0