0

Our team has encountered an issue with one of our VM instance hosted in region Asia-East zone C.

Problem we have encountered: The root partition of the VM instance becomes full. Rebooting the server decreases the disk usage back to 11-15 % utilization.

We tried creating a new VM instance in the same zone but we experienced this problem on the new machine as well. The new machine (Ubuntu 18.04 LTS) is also in the same zone.

After analyzing all the directories in that partition, the sum does not add up to the total available space on the partition.

Appreciate any help in this case.

KayDee
  • 1
  • Are you using docker? I also experience the same problem with the growing size of the docker log too. – Fony Lew May 27 '20 at 15:43
  • Do you install any pkg just after creating the instance? Could you please give me the output of the command 'du -sh /*'? – Mahboob May 27 '20 at 17:00

2 Answers2

0

Look at the sizes of directories in your root volume to see what is large. ncdu -x / is good at this, as you can navigate the tree from / to where the space is being used.

Common files that can grow without bound include text log files and database journals.

Rebooting getting space back often means files were deleted without closing them. Review how files are rotated and purged, namely logrotate configurations.

John Mahowald
  • 32,050
  • 2
  • 19
  • 34
0

Another quickest solution to solve the out of disk space problem is to increase the size of the disk. In general, the partition table allows up to 2 TB in size. Any size more than that will be in vain. Luckily, GCE allows us to grow the disk size without turning off nor restarting. You can do it on the fly by following this guide: https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_partitions.

I know that this suggestion doesn't help tackle the root cause. The usage of ncdu mentioned earlier is a very useful tool to figure out the culprit file. But I think it may help people who are in need of growing the disk size :) Then, solving the problem from the root cause is the best way to go.

Fony Lew
  • 283
  • 1
  • 7