4

I'm new to hadoop.

I have installed my hbase setup using Cloudera (normal installation) on 5 servers. I created some table and filled some data.

Now I monitored the hdfs folder sizes. I can see that the data node is keeping giving consistent reading. But my namenode and journal node disk usage in increasing each time I check.

Though I have a small amount of data, the journal node and name nodes are increasing by 30MB(approx) / day.

Am I missing something in the configurations?

slm
  • 15,396
  • 12
  • 109
  • 124
Bijesh CHandran
  • 477
  • 1
  • 8
  • 22
  • You should go dig around the folders these files reside with `du -sh` and figure out what is taking up the space. It could be logs, image/edits files, or a whole slew of things. – Donald Miner Nov 22 '13 at 16:52
  • I'd agree with Donald, check the size of the logs generated each day – Chris White Nov 22 '13 at 22:00
  • logs are going to a seperate folder. I can see that its the active name node server and the journal nodes adding up the space. I did some research and found that whenever there are some edits in the name node, journal takes up and then secondary name node reads it from journal . But here im not doing any operation in my hbase. – Bijesh CHandran Nov 24 '13 at 11:42
  • When i checked the folders, the edit logs are coming up every 2 mins, that's much expected. But how the Name node makes edit logs, when im not doing any operations on it. – Bijesh CHandran Nov 26 '13 at 13:22
  • I checked the folders and found that edit logs are created every 2 mins in both name node and journal node folders . now one thing i dnt understand is that if my application is not making any operation , then who is editing the namespace metdata? . I can also see that, there are edit logs from the day i created these instances. Is there any hadoop configuration to clear this logs after some time or do i need to delete it manually ? – Bijesh CHandran Nov 26 '13 at 13:25

1 Answers1

1

And after some research i found out the issue, why the edit logs are not cleared. Its a setting in hadoop.

*dfs.namenode.num.extra.edits.retained = 1000000.

This is the default value set . Reference

Bijesh CHandran
  • 477
  • 1
  • 8
  • 22