Write a binary tree in hadoop FS

Question

I need to write a binary tree to HDFS, which i will use to represent a desicion tree. But in order to do that i first need to create a BinaryTreeNode class, which will be the tree node. These are my class attributes:

private String name;
private String attribute;
private String attType;
private String condition;
private String lines;
private BinaryTreeNode leftChild;
private BinaryTreeNode rightChild;

So now i need to implement the write and readFields methods for reading and writing these nodes. These are what i have done:

public void write(DataOutput d) throws IOException 
{
    d.writeUTF(name);
    d.writeUTF(attribute);
    d.writeUTF(attType);
    d.writeUTF(condition);
    d.writeUTF(lines);
    //SOMETHIN FOR LEFT AND RIGHT CHILD
}

public void readFields(DataInput di) throws IOException 
{
    name=di.readUTF();
    attribute=di.readUTF();
    attType=di.readUTF();
    condition=di.readUTF();
    lines=di.readUTF();
    //SOMETHIN FOR LEFT AND RIGHT CHILD
}

BinaryTreeNode read(DataInput in) throws IOException
{
     BinaryTreeNode ob = new BinaryTreeNode();
     ob.readFields(in);
     return ob;
}

What i cant think of is how to write and read my 2 child nodes. Note that the tree is going to be build recursively and every node will have 0-2 childs. So my later purpose is to have a BinaryTree class that will have the attribute BinaryTreeNode root. Thanks

So it is a binary search tree. You can't construct a binary search tree from one traverse. You need to write both `in-order, post-order` or `in-order, pre-order` traverse, so you can construct the main tree. — Majid Azimi, Dec 24 '11 at 22:15
All i need is a way to save and load my tree. If i somehow find a way to save and load my root as an object, my goal will be fullfiled. Well it is not a binary search tree. It is a desicion tree used in machine learning. Since i am making binary splits, it is a binary tree. I need to save and load that model. — jojoba, Dec 24 '11 at 22:29

score 2 · Answer 1 · edited Jun 20 '20 at 09:12

2

I need to write a binary tree to HDFS

All i need is a way to save and load my tree.

What's the reason to go with HDFS? HDFS is a distributed file system on which any type of data/files can be stored. You have write a lot of code to store and retrieve the graphs effectively on large scale.

You can store and retrieve graphs from graph oriented databases like OrientDB and Neo4j.

Also, there are open source frameworks like Apache Giraph, Apache Hama and GoldenOrb. There might also be bindings to interact from Java programs.

edited Jun 20 '20 at 09:12

Community

1
1

answered Dec 25 '11 at 07:21

Praveen Sripati

32,799
16
80
117

I am trying to implement this http://www.bayardo.org/ps/vldb2009.pdf . I need to sent to each mapper the current constructed desicion tree, known as the MFile. I will take a look of those you sent me . Thanks. Although i think it is much simpler. I managed yesterday to save the root of my tree, which had 2 childs. I am still working with it. I want something easy, and not something fast and effective. Thanks :D – jojoba Dec 25 '11 at 12:53

jojoba · Accepted Answer · 2011-12-27T17:27:02.973

Ok i found the solution. I dont know if it is efficient enough but it works. Here is what i did. These are in my class definition:

    public void write(DataOutput d) throws IOException
    {
      d.writeUTF(name);
      d.writeUTF(attribute);
      d.writeUTF(attType);
      d.writeUTF(condition);
      d.writeUTF(lines);
      ObjectWritable left=new ObjectWritable(BinaryTreeNode.class,leftChild);
      left.write(d);
      ObjectWritable right=new ObjectWritable(BinaryTreeNode.class,rightChild);
      right.write(d);
    }

    public void readFields(DataInput di) throws IOException
    {
      name=di.readUTF();
      attribute=di.readUTF();
      attType=di.readUTF();
      condition=di.readUTF();
      lines=di.readUTF();
      leftChild=(BinaryTreeNode) ObjectWritable.readObject(di, new Configuration());
      rightChild=(BinaryTreeNode) ObjectWritable.readObject(di, new Configuration());

    }

And this is how i use them:

For writing:

      BinaryTreeNode bla=new BinaryTreeNode();
      //set the attributes
      ObjectWritable obj=new ObjectWritable(BinaryTreeNode.class,bla);
      obj.write(dos);

For reading:

      BinaryTreeNode bla=new BinaryTreeNode();
      bla= (BinaryTreeNode) ObjectWritable.readObject(in, conf);

This works fine. Now i am able to create my decision tree and store it just by storing my root. :D

Your readFields & Write function implementation is not looking proper. Instead of name=di.readUTF(); you have to use name.readUTF(di) — Bhavesh Gadoya, Oct 25 '15 at 16:15

Write a binary tree in hadoop FS

2 Answers2