1

I have an Hadoop cluster that is quickly running out of room. This cluster is built on RHEL7 VMs on GCP (Google Cloud Platform) Compute Engine. I had originally provisioned the cluster with 10Gb drives on 4 nodes, but that doesn't seem to be enough as Ambari is telling me HDFS is 86% full even though I have yet to load any data. Anyways, I went into the GCP console and 'expanded' my drives to 20Gb on each node. I guess I have two questions- 1) How do I mount the additional capacity in RHEL? 2) How do I tell HDFS/Ambari to use the new capacity?

When I run 'fdisk -l', I can see the expanded drive fdisk -l But I don't see it in df -h df -h Which makes sense, since it's not been mounted yet.

My confusion comes in with fdisk. All of the 20Gb shows up under /dev/sda1, but df only shows 10Gb. How do I mount the extra capacity if fdisk already shows the capacity as being partitioned? Since I just grew an existing disk instead of adding a new one, does that mess things up?

findmnt fdisk

Community
  • 1
Michael Wong
  • 11
  • 2
  • I've no idea about HDFS, but if you expanded the disks, you need to also expand the partitions on it and later the file systems. Please edit your post to include the output of `fdisk -l` and `findmnt` (and use the formatting tools). – Sven May 24 '18 at 18:45
  • Most cloud providers install cloud-init in the OS images they supply and a reboot will automatically increase the partitions and file systems to make use of an increased disk with any further operator input – HBruijn May 24 '18 at 18:48

2 Answers2

0

The partition has been expanded, but not the filesystem on top of it. Use the appropriate command to expand it. With ext4 (and a few others) you would use the resizefs command. I'm not sure about hdfs.

lsd
  • 1,673
  • 10
  • 10
0

The following steps on Resizing the file system and partitions on a persistent disk should guide you through the process of resizing the PD.

Note: After the fdisk command session (which makes changes to partition table), and before re-read partition table, use the partprobe command to notify the Kernel about the change as follows: 

demo@google-my-disk:~$ sudo partprobe /dev/sda1

and confirm changes: 

demo@google-my-disk:~$ sudo fdisk -l

And as per "

How you can tell HDFS/Ambari to use the new capacity?

Start by identifying the disk with the file system and the partition using the following command: 

$ df -h

You need to increase /home space. Steps by steps instruction can be found on the following post on How to increase the capacity of HDFS

Sunny J
  • 607
  • 3
  • 14