5

I'm trying to integrate hadoop with intel lustre. I have added hadoop-lustre-plugin-3.1.0 to hadoop-2.7.3/lib/native folder. Lustre is mounted at /mnt/lustre. I'm getting following error when I start hadoop using start-all.sh

[root@master hadoop]# start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
17/04/06 17:36:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on [ ]
...

core-site.xml :

<property>
  <name>fs.defaultFS</name>
  <value>lustre:///</value>
</property>
<property>
  <name>fs.lustre.impl</name>
  <value>org.apache.hadoop.fs.LustreFileSystem</value>
</property>
<property>
  <name>fs.AbstractFileSystem.lustre.impl</name>
  <value>org.apache.hadoop.fs.LustreFileSystemlustre</value>
</property
<property>
  <name>fs.lustrefs.mount</name>
    <value>/mnt/lustre/hadoop</value>
     <description>This is the directory on Lustre that acts as the root level for Hadoop services</description>
 </property>
<property>
  <name>lustre.stripe.count</name>
  <value>1</value>
</property>
<property>
  <name>lustre.stripe.size</name>
  <value>4194304</value>
</property>
<property>
  <name>fs.block.size</name>
  <value>1073741824</value>
</property>

maprd-site.xml

<property>
    <name>mapreduce.job.map.output.collector.class</name>
    <value>org.apache.hadoop.mapred.SharedFsPlugins$MapOutputBuffer</value>
</property>
<property>
    <name>mapreduce.job.reduce.shuffle.consumer.plugin.class</name>
    <value>org.apache.hadoop.mapred.SharedFsPlugins$Shuffle</value>
</property>

hdfs-site.xml

<property>
<name>dfs.name.dir</name>
<value>/mnt/lustre/hadoop/hadoop_tmp/namenode</value>
<description>true</description>
</property>

Is there any configuration that I have missed in configuration files?

Emre Sevinç
  • 8,211
  • 14
  • 64
  • 105
Trupti
  • 91
  • 2
  • Do you need transparent access to Lustre (e.g. using hdfs:// URIs but reaching Lustre in the background)? Otherwise, "Start the YARN service. - No HDFS necessary. AFS uses Lustre in this case." according to the following source: http://cdn.opensfs.org/wp-content/uploads/2014/04/D3_S28_RunHadoopMapReduceJobsonLustre.pdf – Emre Sevinç Apr 13 '17 at 12:38
  • I have followed steps in pdf and submitted job using yarn but getting error as: – Trupti Apr 17 '17 at 12:04
  • `[@master ~]# yarn jar /opt/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /mnt/lustre/file /mnt/lustre/hadoop/out 17/04/17 17:21:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform.using builtin-java classes where applicable java.lang.SecurityException: class "org.apache.hadoop.fs.LustreFileSystem"'s signer information does not match signer information of other classes in the same package` – Trupti Apr 17 '17 at 12:08

1 Answers1

0

As fs.defaultFS holds the lustre specific URI, the startup script is unable to determine the host in which Namenode has to be started.

Add this property in hdfs-site.xml,

<property>
   <name>dfs.namenode.rpc-address</name>
   <value>namenode_host:port</value>
</property>
franklinsijo
  • 17,784
  • 4
  • 45
  • 63
  • Updated hdfs-site.xml still namenode is not started.Logs are as follows: `$less hadoop-root-namenode-master.tcs.com.log2017-04-17 14:37:24,849 ERROR org.apache.hadoop.hdfs.server.common.Storage: Failed to acquire lock on /mnt/lustre/hadoop/namedir/in_use.lock. If this storage directory is mounted via NFS, ensure that the appropriate nfs lock services are running. java.io.IOException: Function not implemented at sun.nio.ch.FileDispatcherImpl.lock0(Native Method)` – Trupti Apr 17 '17 at 12:20
  • yes. I tried to start hadoop after successfully formatting namenode, still not working. `[root@master ~]# cd /mnt/lustre/hadoop/namedir/current [root@master current]# ll total 16 -rw-r--r-- 1 root root 350 Apr 17 14:36 fsimage_0000000000000000000 -rw-r--r-- 1 root root 62 Apr 17 14:36 fsimage_0000000000000000000.md5 -rw-r--r-- 1 root root 2 Apr 17 14:36 seen_txid -rw-r--r-- 1 root root 203 Apr 17 14:36 VERSION` – Trupti Apr 18 '17 at 05:26
  • @Trupti Try changing the `dfs.name.dir` to some directory which is not lustreFS. Reformat and Restart the services. – franklinsijo Apr 18 '17 at 06:43