webhdfs two steps upload a file

Question

I build a hadoop cluster with 4 machines:

• {hostname}: {ip-address}
• master: 192.168.1.60
• slave1: 192.168.1.61
• slave2: 192.168.1.62
• slave3: 192.168.1.63

I use HttpFS upload a file to hdfs with restful way, there contains two steps to finish the task.

• Step 1: Submit a HTTP POST request without automatically following redirects and without sending the file data.

  curl -i -X POST "http://192.168.1.60:50070/webhdfs/v1/user/haduser/myfile.txt?op=APPEND"

the server return result like:

Location:http://slave1:50075/webhdfs/v1/user/haduser/myfile.txt?op=CREATE&user.name=haduser&namenoderpcaddress=master:8020&overwrite=false

• step 2: use the response address to upload the file.

In step 1, How could I get the datanode's ip address(192.168.1.61) rather than the hostname (slave1)?

Answer 1

If your hadoop version>=2.5, at every datanode config ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml file. add: property dfs.datanode.hostname, the value is datanodes's ip address.