0

Ignite nodes are not getting connected in kubernetes cluster. I have used API method TcpDiscoveryKubernetesIpFinder and set kubernetes master Ip address explicitly as https://192.167.120.92:443 (ip address of kubernetes master node) using ipFinder.setMasterUrl("https://192.167.120.92:443").Following is the error log I am getting and ignite ipfinder is unable to retrieve ip addresses of ignite pods.

[11:25:43,471][INFO][main][IgniteKernal]

>>>    __________  ________________
>>>   /  _/ ___/ |/ /  _/_  __/ __/
>>>  _/ // (7 7    // /  / / / _/
>>> /___/\___/_/|_/___/ /_/ /___/
>>>
>>> ver. 1.9.0#20170302-sha1:a8169d0a
>>> 2017 Copyright(C) Apache Software Foundation
>>>
>>> Ignite documentation: http://ignite.apache.org

[11:25:43,473][INFO][main][IgniteKernal] Config URL: n/a
[11:25:43,473][INFO][main][IgniteKernal] Daemon mode: off
[11:25:43,473][INFO][main][IgniteKernal] OS: Linux 3.10.0-514.21.2.el7.x86_64 amd64
[11:25:43,474][INFO][main][IgniteKernal] OS user: root
[11:25:43,477][INFO][main][IgniteKernal] PID: 7
[11:25:43,478][INFO][main][IgniteKernal] Language runtime: Java Platform API Specification ver. 1.8
[11:25:43,478][INFO][main][IgniteKernal] VM information: OpenJDK Runtime Environment 1.8.0_111-8u111-b14-2~bpo8+1-b14 Oracle Corporation OpenJDK 64-Bit Server VM 25.111-b14
[11:25:43,480][INFO][main][IgniteKernal] VM total memory: 27.0GB
[11:25:43,481][INFO][main][IgniteKernal] Remote Management [restart: off, REST: on, JMX (remote: off)]
[11:25:43,481][INFO][main][IgniteKernal] IGNITE_HOME=/opt/ignite/apache-ignite-fabric-1.9.0-bin
[11:25:43,481][INFO][main][IgniteKernal] VM arguments: [-DIGNITE_QUIET=false]
[11:25:43,482][INFO][main][IgniteKernal] Configured caches ['ignite-marshaller-sys-cache', 'ignite-sys-cache', 'ignite-atomics-sys-cache']
[11:25:43,492][INFO][main][IgniteKernal] 3-rd party licenses can be found at: /opt/ignite/apache-ignite-fabric-1.9.0-bin/libs/licenses
[11:25:43,609][INFO][main][IgnitePluginProcessor] Configured plugins:
[11:25:43,609][INFO][main][IgnitePluginProcessor]   ^-- None
[11:25:43,610][INFO][main][IgnitePluginProcessor]
[11:25:43,700][INFO][main][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0, selectorsCnt=28, selectorSpins=0, pairedConn=false]
[11:25:43,707][WARNING][main][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.
[11:25:43,757][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
[11:25:43,802][WARNING][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival).
[11:25:43,807][WARNING][main][NoopSwapSpaceSpi] Swap space is disabled. To enable use FileSwapSpaceSpi.
[11:25:43,809][INFO][main][IgniteKernal] Security status [authentication=off, tls/ssl=off]
[11:25:44,249][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211]
[11:25:44,316][INFO][main][IgniteKernal] Non-loopback local IPs: 172.17.0.3, fe80:0:0:0:42:acff:fe11:3%eth0
[11:25:44,316][INFO][main][IgniteKernal] Enabled local MACs: 0242AC110003
[11:25:44,380][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port [port=47500, localHost=0.0.0.0/0.0.0.0, locNodeId=57a9922d-0b32-455f-ab6a-e5459a83a02a]
[11:25:44,854][SEVERE][main][TcpDiscoverySpi] Failed to get registered addresses from IP finder on start (retrying every 2000 ms).
class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite pods IP addresses.
        at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172)
        at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1613)
        at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1562)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:974)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:837)
        at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:351)
        at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1850)
        at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:268)
        at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:685)
        at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1626)
        at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:924)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1799)
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1602)
        at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1042)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:569)
        at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:516)
        at org.apache.ignite.Ignition.start(Ignition.java:322)
        at ServiceDeployment.main(ServiceDeployment.java:85)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
        at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
        at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
        at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1546)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
        at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:153)
        ... 17 more

I have tried flannel and weave net networks, but in both cases I am getting same error. Also according to post Master-Node communication, by default cluster nodes should be able to communicate to master by https on port 443 and should be able to retrieve ip addresses in my case, but it is not happening.

rishi007bansod
  • 1,283
  • 2
  • 19
  • 45
  • Your node cannot connect to Kubernetes API server (connection is refused). Check if they are visible to each other. – Konstantin Nov 09 '17 at 16:48

1 Answers1

0

Actually, whatever you are doing is correct. But you have to set the path to the token.

ipFinder.setAccountToken(/path/to/token)

By default this would be pointing to /var/run/secrets/kubernetes.io/serviceaccount/token

You can try to give permission to your service account or add it to the admin list. I encountered this issue and just gave this role to the service account.

Run this: 

kubectl create clusterrolebinding serviceaccounts-cluster-admin--clusterrole=cluster-admin --group=system:serviceaccounts
papaya
  • 1,505
  • 1
  • 13
  • 26