0

I am running an Ubuntu Amazon EC2 instance with a 100gb volume mounted. However, when I hit 56gb, the FS says that it has run out of storage. It clearly has plenty of space available. This is the result of the df.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        97G   56G   42G  58% /
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           3.2G  852K  3.2G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/xvda15     105M  5.3M  100M   5% /boot/efi
tmpfs           1.6G  4.0K  1.6G   1% /run/user/1000

As you can see, the OS knows that there is space, but seemingly refuses to write. Prior to experiencing this problem, I had increased the volume size from 8GB to 100GB, and used the commands growpart and resize2fs to scale the volume to full size. I have a feeling that this is due to some unknown volume size limit within either Ubuntu or AWS, but I have no idea how to diagnose or solve this problem. My initial guess was that EXT4 had a limit, but that doesn't seem to be applicable here (the limit is 1TB). Here is the result of lsblk:

$ lsblk
NAME     MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0      7:0    0 25.1M  1 loop /snap/amazon-ssm-agent/5656
loop1      7:1    0 55.5M  1 loop /snap/core18/2409
loop2      7:2    0 61.9M  1 loop /snap/core20/1518
loop3      7:3    0 79.9M  1 loop /snap/lxd/22923
loop4      7:4    0   47M  1 loop /snap/snapd/16010
loop5      7:5    0   47M  1 loop /snap/snapd/16292
xvda     202:0    0  100G  0 disk 
├─xvda1  202:1    0 99.9G  0 part /
├─xvda14 202:14   0    4M  0 part 
└─xvda15 202:15   0  106M  0 part /boot/efi

Any help here would be greatly appreciated, thanks in advance :)

rchurch4
  • 859
  • 6
  • 14
  • 1
    `when I hit 56gb, the FS says that it has run out of storage`: what's the specific error message and where are you getting it? My guess: you're putting something at the wrong mountpoint, not / – Barry Carter Jul 06 '22 at 16:00
  • It occurs in a python script. "OSError: out of storage." I would be surprised if it was being put at the wrong mountpoint. Is it possible that it's overflowing one of the tmpfs mountpoints? – rchurch4 Jul 06 '22 at 16:02
  • 1
    It could be, yes. https://stackoverflow.com/questions/59522627/how-to-set-up-the-temp-dir-path-in-python shows how to tweak the /tmp directory or you can just make /tmp bigger. I assume you have enough data that it's reasonable you'd exceed 8GB of /tmp space – Barry Carter Jul 06 '22 at 16:06
  • I think making tmp bigger will be the right answer. I will check back in a moment once I have tried it :) – rchurch4 Jul 06 '22 at 16:40
  • @barrycarter it looks like the problem occurred when I increased the size of the volume. I think that the /tmp folder was completely unmounted and never remounted. I have remounted the /tmp folder and it SEEMS that I do not have this problem anymore. Further testing is ongoing, but that you for bringing my attention to the /tmp folder. If you want to summarize in an answer I will endorse it :) – rchurch4 Jul 06 '22 at 17:44
  • 1
    Feel free to self-answer, I'm not in it for the points :) Technically, you mentioned tmpfs first – Barry Carter Jul 07 '22 at 12:16
  • Turns out this was not the answer anyway sadly. after mounting the /tmp folder, the problem persisted. I created a new EC2 instance in case changing the volume size was the culprit. The same exact problem occurs again. – rchurch4 Jul 07 '22 at 15:28

0 Answers0