1

I wanted to know who will create the block ids for blocks in hadoop either HDFS client or Name node.Please let me know.

sidhartha pani
  • 623
  • 2
  • 12
  • 23

1 Answers1

2

The NameNode allocates the block ID and gives it to the client. The client then uses this block ID while communicating with a DataNode to write data into the block.

Apache JIRA HDFS-4645 documents the current design for allocation of block IDs. It uses a monotonically increasing ID starting from a specific constant. If you're interested in seeing the code for this, refer to the BlockIdManager and SequentialBlockIdGenerator classes in the Apache Hadoop codebase.

Chris Nauroth
  • 9,614
  • 1
  • 35
  • 39
  • Please correct me If am wrong. As per my understanding client will split the file into blocks. Client asks Name node to get availability of Data nodes Then writes these blocks into Data nodes. Once writing process is done. Data nodes will provide block information to Name node.Then only Name node update meta data In this process where block id is generated as clientis actually splitting the blocks.How Name node assigns block ids for blocks if Name node is not being updated by Data nodes. – sidhartha pani Jan 26 '17 at 07:56
  • 1
    You are correct that the client decides when a new block needs to be allocated for the file, but the NameNode is always the central point for assignment of a new block ID. The client makes an RPC call to the NameNode to allocate the block. Then, the NameNode generates a block ID, persists it to metadata and sends that block ID back to the client. Then, the client uses that block ID when communicating with the DataNode. – Chris Nauroth Jan 26 '17 at 08:07
  • 1
    You mean when client is asking Name node for allocating Data nodes for a block it sends block id for that block to the Client ? – sidhartha pani Jan 26 '17 at 14:22
  • @sidharthapani yes, that's it exactly! The NameNode gives the client the newly assigned block ID. The client then passes that block ID to the DataNode when it writes data to the block. – Chris Nauroth Jan 26 '17 at 17:21
  • I understand now thanks a lot – sidhartha pani Jan 27 '17 at 03:57