how to re balance the HDFS data size on data node disks

Asked Jul 24 '19 at 22:01

Active Jul 25 '19 at 04:08

Viewed 194 times

we have production cluster with HDP - 2.6.4 version

we have 186 data-node machines ( DELL MACHINES WITH 10 disks )

we try to re balance the data on the disks so disks will be with the same used size but without success

we feel that 2.6.4 version not have the tools that support re balance!!!

as I mentioned on each data-node machine we have 10 disks while each disk is 1.8T

and some of the disks are 55% used

and some of them are only 1% used

so we have non balanced disks ( its like some disk are not useful ) , but why HDFS not balanced the data on all disks??

my question - from which HDP version , we can re balance the data-node disks ?

Dose 2.6.5 version support re balance ?

or from 3.X ?

please advice , what we can do ?

as I mentioned this is very huge cluster and

we get the bad feeling that the current HDP version ( 2.6.4 ) not support any re balance - is it true?

edited Jul 25 '19 at 04:08

OneCricketeer

asked Jul 24 '19 at 22:01

Judy

@Dear Judy , can you please show us one of the datanode machines with the disks? – King David Jul 24 '19 at 22:12
I think @cricket_007 can answer on your question , your vendor is hoprtonworks , so why not ask them? – King David Jul 24 '19 at 22:14
Hadoop 2 does not have any internode rebalancing. I'm not aware of any alternative – OneCricketeer Jul 25 '19 at 04:08
What is `dfs.datanode.fsdataset.volume.choosing.policy` set to? – tk421 Jul 25 '19 at 16:52

0 Answers0