2

I'm trying to setup hadoop on mesos using the document below:

https://docs.mesosphere.com/tutorials/run-hadoop-on-mesos/

I'm facing a problem on step-9

sudo -u mapred ./hadoop-2.0.0-mr1-cdh4.2.1/bin/hadoop dfs -rm -f /hadoop-2.0.0-mr1-cdh4.2.1.tgz
sudo -u mapred /usr/bin/hadoop dfs -copyFromLocal ./hadoop-2.0.0-mr1-cdh4.2.1.tgz /

I am still new to this concept. I have to configure a mesos cluster using this tutorial:

https://www.digitalocean.com/community/tutorials/how-to-configure-a-production-ready-mesosphere-cluster-on-ubuntu-14-04

Now I'm getting errors while performing dfs commands:

root@station1:~# sudo -u mapred ./hadoop-2.0.0-mr1-cdh4.2.1/bin/hadoop dfs -rm -f /hadoop-2.0.0-mr1-cdh4.2.1.tgz
-rm: Expected authority at index 7: hdfs://
Usage: hadoop fs [generic options] -rm [-f] [-r|-R] [-skipTrash] <src> ...
JSON C11
  • 11,272
  • 7
  • 78
  • 65
Shiva.manikandn
  • 21
  • 1
  • This sounds to me that you have a problem with `core-site.xml`, that is, the values for `fs.defaultFS` and/or `fs.default.name` are not set correctly. Can you check again and make sure 100% that you've followed the guide in step 8? – Michael Hausenblas Apr 11 '15 at 11:02

1 Answers1

1

This tutorial assumes you have HDFS already installed on your cluster. You can do this by manually installing HDFS on each node, or you can try out the new HDFS framework: https://github.com/mesosphere/hdfs

Does hadoop fs -ls hdfs:// work on its own? If not, you'll need to install and configure HDFS appropriately.

Adam
  • 4,322
  • 1
  • 16
  • 22
  • i have setup HDFS package which you menstion above https://github.com/mesosphere/hdfs. but, having problem to start this HDFS service .it saying unable to allocate resource like error am getting . Please help me to slove this . – Shiva.manikandn Apr 18 '15 at 14:18
  • Could you provide the steps you used and the exact error/logs you got? You will need at least 4 Mesos slaves for the HDFS framework to start up NameNodes, JournalNodes, and at least one DataNode. – Adam Apr 20 '15 at 05:05
  • Thanks for response . This is what error am getting once am stating the HDFS service on mesos master node.. root@master1:/opt# cd hdfs-mesos-0.1.0/ root@master1:/opt/hdfs-mesos-0.1.0# ./bin/hdfs-mesos 15:42:52.100 [Thread-35] INFO org.apache.mesos.hdfs.Scheduler - Received 1 offers 15:42:52.105 [Thread-35] INFO org.apache.mesos.hdfs.Scheduler - Offer does not have enough resources – Shiva.manikandn Apr 20 '15 at 10:20
  • You're either going to need larger Mesos slaves to be able to launch the NNs/DNs, or you'll have to unpack the HDFS-framework tarball, reduce the cpu/memory requirements, and repackage it. – Adam Apr 21 '15 at 16:34
  • I am the beginner to mesos and hadoop.my mesos cluster have 5 slaves and 1 master, each node configured as (CPU:1,MEM:245,Disk:3.5GB) .This all created in virtual box. I'm not aware whether myself is going in a right path so please do needful. i am requesting you to documentation or step by step . Thanks in Advance. – Shiva.manikandn Apr 23 '15 at 12:40
  • 245MB of memory is tiny, and by default HDFS needs more than this to launch its NN/DN tasks. Even just 1 cpu is not enough per node by default. See the various heap.size and cpus variables in mesos-site.xml: https://github.com/mesosphere/hdfs/blob/master/conf/mesos-site.xml#L100 You'll either want to add more resources to your VMs (1+GB, 2 cpus?), or adjust these config requirements down to something you can actually handle. – Adam Apr 23 '15 at 16:31
  • Thanks for your help Adam. Now, i have installed and configured mesosphere both master and slave in single physical machine (RAM:4GB,DISK:320GB). and then i have successfully setup HDFS package which you mentioned above github.com/mesosphere/hdfs. I'm getting mesosphere web UI journalnode1 is in RUNNING state but, i'm not getting connect web UI for namenode http://10.9.168.7:50070 on browser and terminal "hadoop fs -ls hdfs://10.9.168.76/" getting error "techteam-desktop.xerago.com:8020 failed on connection exception: java.net.ConnectException: Connection refused". Please help me to slove. – Shiva.manikandn Apr 26 '15 at 13:32
  • The HDFS framework for need at least 4 Mesos slaves so it can start up NameNodes, JournalNodes, and at least one DataNode. The HDFS framework first tries to launch 3 JNs, for a redundant metadata/log store, before it will launch even one NameNode. This framework installs a Highly-Available (HA) HDFS cluster, so you need to have more than one node. If you want to run HDFS on a single node (for testing only, because you should really have it on multiple nodes for redundant data storage), then you'll have to install HDFS yourself. – Adam Apr 27 '15 at 05:16
  • i have cluster with 2 physical and 2 virtual machine (slave1/master: cpu=4,mem=2.7GB,Disk=178.2 GB,type=physical|slave2: cpu=1,mem=1001MB,Disk=2.5 GB,type=virtual |slave3: cpu=1,mem=1001MB,Disk=2.5 GB,type=virtual |slave4: cpu=4,mem=6.7GB,Disk=77.4 GB,type=physical ). this time geting error while am trying to start HDFS package in master (./bin/hdfs-mesos) using this command. "Failed to fetch: http://127.0.0.1:8765/hdfs-mesos-0.1.0.tgz Failed to synchronize with slave (it's probably exited)" – Shiva.manikandn May 04 '15 at 13:51
  • started journalnode1 with master/slave sucessfully but, it trying to start in slave machine it getting error i think it pointing localhost ip of master machine. – Shiva.manikandn May 04 '15 at 13:52
  • For the 127.0.0.1 fetch error, check the `/etc/hosts` setup on your slave to make sure that the output of `hostname` maps to something other than the loopback IP. Alternatively, you can override Mesos' hostname/ip resolution by specifying `mesos-slave --ip=...` – Adam May 05 '15 at 05:29
  • Thanks for your help Adam. i made correction `/etc/hosts` file regarding local ip . now launching task in slave machine too. but, two journalnode are started in different physical machines. but, still NameNodes and DataNodes not yet started . – Shiva.manikandn May 05 '15 at 12:12
  • I'm guessing the two JNs started on the two 4-cpu nodes. The default resource requirements for these tasks are rather large, so you might not be able to fit a JN on a 1-cpu, 1GB-RAM node. And since HDFS needs all three JNs to start, it's waiting for an appropriately sized offer to launch the third JN before it can launch any NNs/DNs. You can either use bigger nodes, or change the default heapsize/cpu config: https://github.com/mesosphere/hdfs/blob/master/conf/mesos-site.xml#L92 – Adam May 06 '15 at 15:33
  • i have setup cluster with 7 node each (cpu:4,Mem: > 1.9 GB). Now JNs and zkfc RUNNING in three machine but, NNs not started So, i have fine turned the setting of mesos-site.xml file downgrade namenode.heap.size 4GB to 1GB . Now, namenode1 is stated in one machine and namenode2 got FAILED. still DN not started. Error 1: `We need to coloate the namenode with a journalnode and there isno journalnode running on this host. 10.0.2.39` Error 2: `Received status update for taskId=task.namenode.namenode.NameNodeExecutor.1430931455570 state=TASK_FAILED message='' stagingTasks.size=0` – Shiva.manikandn May 06 '15 at 17:29
  • Error 1 just shows up when you're trying to launch a NN, but you get an offer from a non-JN host. It should be fine, since there are other nodes where the JN is running, and Myriad should also get offers from them. – Adam May 07 '15 at 15:24
  • Error 2 is more relevant. The second NNExecutor failed. Please use the Mesos UI to navigate to that task's sandbox and see what its stdout/stderr reports. – Adam May 07 '15 at 15:25