Running out of disk space in Amazon EC2, can't find what I am using my storage for

Question

I am Running an AWS ami using a T2.large instance using the US East. I was trying to upload some data and I ran in the terminal:

df -h

and I got this result:

Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           799M  8.6M  790M   2% /run
/dev/xvda1      9.7G  9.6G   32M 100% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           799M     0  799M   0% /run/user/1000

I know I have not uploaded 9.7 GB of data to the instance, but I don't know what /dev/xvda1 is or how to access it.

I also assume that all the tmpfs are temporal files, how can I erase those?

Answering some of the questions in the coments, I runned

sudo du -sh /*

And I got:

16M /bin
124M    /boot
0   /dev
6.5M    /etc
2.7G    /home
0   /initrd.img
0   /initrd.img.old
4.0K    /jupyterhub_cookie_secret
16K /jupyterhub.sqlite
268M    /lib
4.0K    /lib64
16K /lost+found
4.0K    /media
4.0K    /mnt 
562M    /opt
du: cannot access '/proc/15616/task/15616/fd/4': No such file or directory
du: cannot access '/proc/15616/task/15616/fdinfo/4': No such file or directory
du: cannot access '/proc/15616/fd/4': No such file or directory
du: cannot access '/proc/15616/fdinfo/4': No such file or directory
0   /proc
28K /root
8.6M    /run
14M /sbin
8.0K    /snap 
8.0K    /srv
0   /sys
64K /tmp
4.7G    /usr
1.5G    /var
0   /vmlinuz
0   /vmlinuz.old

Xvda1 is the disk drive mounted to the VMs file system. It definitely suggests something in installed on the ami that is consuming a large amount of data. It's the filesystem. You definitely have an excessively large amount of files somewhere on the filesystem. You can see it's mounted to '/'. — Dandy, Jun 14 '18 at 00:04

score 9 · Answer 1 · answered Jun 14 '18 at 14:01

9

When you run out of root filesystem space, and aren't doing anything that you know consumes space, then 99% of the time (+/- 98%) it's a logfile. Run this:

sudo du -s /var/log/* | sort -n

You'll see a listing of all of the sub-directories in /var/log (which is the standard logging destination for Linux systems), and at the end you'll probably see an entry with a very large number next to it. If you don't see anything there, then the next place to try is /tmp (which I'd do with du -sh /tmp since it prints a single number with "human" scaling). And if that doesn't work, then you need to run the original command on the root of the filesystem, /* (that may take some time).

Assuming that it is a logfile, then you should take a look at it to see if there's an error in the related application. If not, you may just need to learn about logrotate.

answered Jun 14 '18 at 14:01

guest

91
1

/opt/python/log is the log folder for instances using Python – Greg Holst Aug 03 '20 at 08:13
can we delete the log file. Did it impact on anything running on server? I'm having this output (last entries of command) `28 /var/log/syslog 28 /var/log/syslog.1 52 /var/log/unattended-upgrades 68 /var/log/amazon 100 /var/log/apt 312 /var/log/cloud-init.log 408 /var/log/auth.log.2.gz 476 /var/log/auth.log.4.gz 1764 /var/log/nginx 1968 /var/log/auth.log 4416 /var/log/btmp 4828 /var/log/auth.log.1 24832 /var/log/btmp.1` – Lalitesh Upadhyaya Apr 06 '21 at 23:11
/var/log/journal was quite large for me after running this command, so I reduced the size by running sudo journalctl --vacuum-size=100M as seen https://askubuntu.com/questions/1238214/big-var-log-journal – Dylan w Feb 23 '22 at 07:34

Brandon Miller · Accepted Answer · 2018-06-14T09:53:49.123

3

/dev/xvda1 is your root volume. The AMI you listed has a default root volume size of 20GB as you can see here:

Describe the image and get it's block device mappings:

aws ec2 describe-images --image-ids ami-3b0c205e --region us-east-2 | jq .Images[].BlockDeviceMappings[]

Look at the volume size

{
  "DeviceName": "/dev/sda1",
  "Ebs": {
    "Encrypted": false,
    "DeleteOnTermination": true,
    "VolumeType": "gp2",
    "VolumeSize": 20,
    "SnapshotId": "snap-03341b1ff8ee47eaa"
  }
}
{
  "DeviceName": "/dev/sdb",
  "VirtualName": "ephemeral0"
}
{
  "DeviceName": "/dev/sdc",
  "VirtualName": "ephemeral1"
}

When launched with the correct volume size of 20GB there is plenty of free space (10GB)

root@ip-10-100-0-64:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            488M     0  488M   0% /dev
tmpfs           100M  3.1M   97M   4% /run
/dev/xvda1       20G  9.3G   11G  49% /
tmpfs           496M     0  496M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           496M     0  496M   0% /sys/fs/cgroup
tmpfs           100M     0  100M   0% /run/user/1000

It appears the issue here is the instance was launched with 10GB (somehow, I didn't think this was possible) of storage instead of the default 20GB.

edited Jun 14 '18 at 09:53

answered Jun 14 '18 at 00:25

Brandon Miller

4,695
1
20
27

Where do you see ami-3b0c205e? It isn't possible to launch an instance with smaller volumes than the AMI's associated snapshots, so that can't be the source image for this instance. – Michael - sqlbot Jun 14 '18 at 09:23
D'oh, you're absolutely correct. I should have checked the edit history. But I think OP must have been mistaken on that point. It isn't possible for the volumes to be smaller, because EBS won't allow it. Creating a volume from a snapshot can only ever create a volume the same size or larger, and EBS doesn't support making existing volumes smaller. (Wouldn't that be fun to support? What could possibly go wrong?) – Michael - sqlbot Jun 14 '18 at 09:56
@Michael-sqlbot I didn't think it was possible either, and trying to replicate launching with 10 GB gives me the error you would expect. The ami was in the question before it was editted. So, either the asker gave the wrong ami or managed to launch with 10 GB. In any case, you can clearly see the default usage in this case is around 9.3 GB and should help the asker to realize they didn't upload 9.7 GB. – Brandon Miller Jun 14 '18 at 09:56
1

@Michael-sqlbot Updated my answer to reflect that **I** can't seem to do the same thing, so not 100% sure exactly how they pulled it off. – Brandon Miller Jun 14 '18 at 09:58
1

@Michael-sqlbot so what I did was to get a reserved instance, (I am very new to this and the instance was aquired by my company). I didn't know anything about the Elastic block Store that had the storage of my EC2, (still getting used to this). Now I see that with the reserved instance we get a a 10 gb EBS volume which will be free for 1 year as part of the Free Tier – Derek Corcoran Jun 14 '18 at 16:06

score 0 · Answer 3 · answered Jun 14 '18 at 00:04

/dev/xvda1 is your disk based storage on the amazon storage system.

It is the only storage on your system have, and it contains your operation system and all data. So I guess most of the space it used by the Ubuntu installation

Remember: T instances at amazon don't have any local disk at all.

Running out of disk space in Amazon EC2, can't find what I am using my storage for

3 Answers3

Describe the image and get it's block device mappings:

Look at the volume size

When launched with the correct volume size of 20GB there is plenty of free space (10GB)