1

I am trying to open the following path in HDFS:

TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/user/flume/tweets

I opened a new browser, and I typed http://localhost:50070/dfshealth.html#tab-overview

I get the following error:

There are 2 missing blocks.

The following files may be corrupted:

blk_1073742237 /hbase/data/hbase/meta/1588230740/info/c5da7e591d294ae58968f4d0f2e8ffd9 blk_1073742231 /hbase/WALs/quickstart.cloudera,60020,1482726320014-splitting/quickstart.cloudera%2C60020%2C1482726320014..meta.1482726370496.meta

It is saying how to find possible solution for this, but is there any simplified way of solving this problem?

Community
  • 1
  • 1
Joseph Desire
  • 23
  • 2
  • 9

1 Answers1

0

This might be helpful:

Check the corrupted blocks using the command:

hdfs fsck <path> -list-corruptfileblocks
e.g. hdfs fsck /hbase -list-corruptfileblocks

Move the corrupted blocks to /lost+found using:

hdfs fsck <path> -move
e.g. hdfs fsck /hbase -move

OR Delete the corrupted blocks using:

hdfs fsck <path> -delete
e.g. hdfs fsck /hbase -delete

Sometimes you'll be asked for Superuser privileges, in which case, append sudo -u hdfs before your command e.g. sudo -u hdfs hdfs fsck /hbase -list-corruptfileblocks

PradeepKumbhar
  • 3,361
  • 1
  • 18
  • 31
  • Thank you so much for your help, i could delete them. But i still dont get the following: Open the Mozilla browser in your VM, and go to /user/flume/tweets in HDFS. It is the step 24 on this link: https://www.eduonix.com/blog/bigdata-and-hadoop/flume-installation-and-streaming-twitter-data-using-flume/ – Joseph Desire Jan 31 '17 at 12:48
  • Does not matter if you are using a VM or directly the machine, the meaning of the 24th step is: Open your browser and type in URL as `http://localhost:50070/explorer.html`. Then navigate to `user >flume >tweets` directory – PradeepKumbhar Feb 01 '17 at 06:10
  • The port in my case is 50070. It could be different in your case. Check this: http://stackoverflow.com/questions/22855232/default-namenode-port-of-hdfs-is-50070-but-i-have-come-across-at-some-places-802 – PradeepKumbhar Feb 01 '17 at 06:14
  • Thanks once more Daemon! I did that and i got the same localhost as you. But this is the result i got: Browse Directory Path does not exist on HDFS or WebHDFS is disabled. Please check your path or enable WebHDFS What can be the problem please_ – Joseph Desire Feb 01 '17 at 06:54
  • In step23,command: flume-ng agent -n TwitterAgent -c conf -f /usr/lib/apache-flume-1.7.0-bin/conf/flume.conf. The reult is:17/01/31 12:04:08 INFO source.DefaultSourceFactory: Creating instance of source Twitter, type com.cloudera.flume.source.TwitterSource 17/01/31 12:04:08 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows. org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource. Maybe the problem is here? I dont understand please, can you help? – Joseph Desire Feb 01 '17 at 07:04
  • The exception you are getting is something different from the original question. Please accept/upvote the above answer if it solves your original problem and create a new question for FlumeException. – PradeepKumbhar Feb 01 '17 at 07:14