Hadoop connection refused to ResourceManager on localhost:8032

Question

I'm new to Hadoop, and have been trying to get it working on my machine since the last several hours. I've mostly followed the installation and setup instructions from these guides - 1, 2, 3, 4, 5, 6.

I have also read this SO question on Hadoop: Connecting to ResourceManager failed several times, but nothing has helped.

Here are my configuration files:

core-site.xml -

<configuration>
<property>
   <name>hadoop.tmp.dir</name>
   <!-- <value>/tmp/hadoop-${user.name}</value> -->
   <value>/app/hadoop/tmp</value>
   <description>A base for other temporary directories.</description>
 </property> 
<property>
   <name>fs.default.name</name>
   <value>hdfs://localhost:9000</value>
 </property>
</configuration>

hdfs-site.xml -

<configuration>

 <property>
     <name>dfs.replication</name>
     <value>1</value>
 </property>
 <property>
  <name>dfs.namenode.name.dir</name>
  <value>${user.home}/hadoop/data/namenode</value>
</property>

<property>
  <name>dfs.datanode.data.dir</name>
  <value>${user.home}/hadoop/data/datanode</value>
</property>
</configuration>

mapred-site.xml -

<configuration>
    <property>
        <name>mapred.job.tracker</name>
        <value>localhost:9001</value>
    </property>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

yarn-site.xml -

<configuration>
    <!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>localhost:8032</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>localhost:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>localhost:8031</value>
    </property>
</configuration>

The setup guides also require making changes in hadoop-env.sh, but the required changes are already present in this case for OSX, so I didn't make any further changes -

case ${HADOOP_OS_TYPE} in
  Darwin*)
    export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.realm= "
    export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.kdc= "
    export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.conf= "
  ;;
esac

In my .zshrc file, I've added the hstart and hstop alias like so -

# hadoop config
alias hstart="/usr/local/Cellar/hadoop/3.1.1/sbin/start-dfs.sh;/usr/local/Cellar/hadoop/3.1.1/sbin/start-yarn.sh"
alias hstop="/usr/local/Cellar/hadoop/3.1.1/sbin/stop-yarn.sh;/usr/local/Cellar/hadoop/3.1.1/sbin/stop-dfs.sh"

I think my hstart command is working normally -

➜  ~ hstart
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [Manish-Macbook-Pro-15.local]
2019-03-01 02:26:13,502 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting resourcemanager
Starting nodemanagers

Here is the output of jps -

➜  ~ jps
30548 NameNode
31172 Jps
30796 SecondaryNameNode
30655 DataNode

After this, I try to run of the sample jar files (hadoop-mapreduce-examples-3.1.1.jar) with a sample job from one of the guides -

➜  ~ cd /usr/local/Cellar/hadoop/3.1.1/libexec/share/hadoop/mapreduce
➜  mapreduce hadoop jar hadoop-mapreduce-examples-3.1.1.jar pi 2 5

This is when the ConnectionRefused error shows up -

Number of Maps  = 2
Samples per Map = 5
2019-03-01 02:26:39,491 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Starting Job
2019-03-01 02:26:42,134 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
2019-03-01 02:26:43,440 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:44,443 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:45,449 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:46,452 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:47,456 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:48,461 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:49,463 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:50,469 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:51,475 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:52,481 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:53,496 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:54,501 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:55,507 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:56,510 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:57,513 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:58,517 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:26:59,520 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:27:00,526 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:27:01,528 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:27:02,530 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2019-03-01 02:27:02,533 INFO retry.RetryInvocationHandler: java.net.ConnectException: Call From Manish-Macbook-Pro-15.local/192.168.200.79 to localhost:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused, while invoking ApplicationClientProtocolPBClientImpl.getNewApplication over null after 1 failover attempts. Trying to failover after sleeping for 31316ms.

After this, it keeps trying to connect repeatedly. I have to open up a new terminal tab to execute hstop.

I have no idea what is causing this error. I've started from scratch with hdfs namenode -format several times, but each time the same error.

A few other observations I have -

• The guides mention that once hadoop is up and running, I can view the status of things (like ResourceManager) in a browser using the corresponding default ports, but I've tried several ports - localhost:50070, localhost:8088 and many more, but nothing works.

• In the yarn-site.xml file, I've used localhost:8032 as the value for <name>yarn.resourcemanager.address</name>, but in my terminal logs, it shows 2019-03-01 02:26:48,461 INFO ipc.Client: Retrying connect to server: localhost/127.0.0.1:8032.. I don't know why it's localhost/127.0.0.1:8032 instead of localhost:8032.

Here are the contents of /etc/hosts -

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1       localhost
255.255.255.255 broadcasthost
::1             localhost

Would appreciate any help with this.