0

I'm having the strangest problem for days now. I took over a WordPress website of a company that was originally developed by another person – the codebase is a mess but I was able to go over it and make sure it at least is working.

The database is huge (70mb) and there is a lot of plugin dependencies on the site.

However the site works generally without issues now and I'm hosting it on an EC2 with a bitnami stack for WordPress.

The weird thing though is that everyday (for instance today morning) I check the site and it's down … 

Service Unavailable The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Additionally, a 503 Service Unavailable error was encountered while trying to use an ErrorDocument to handle the request.

When logging into the server with ssh and trying to restart apache I get this:

Failed to unmonitor apache: write /var/lib/gonit/state: no space left on device Syntax OK /opt/bitnami/apache2/scripts/ctl.sh : apache not running Syntax OK (98)Address already in use: AH00072: make_sock: could not bind to address [::]:80 (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down AH00015: Unable to open logs /opt/bitnami/apache2/scripts/ctl.sh : httpd could not be started Failed to monitor apache: write /var/lib/gonit/state: no space left on device

I had this the 3rd time in 3 days now even though I restored the server from a snapshot with a volume of 200gb (for testing purposes) and all site files including uploads only have 5gb.

The site is running on an EC2 (t2.medium) with 200gb volume now and today morning I can't restart apache. Yesterday evening when restoring from a snapshot the site works well and normal - it's actually even fast.

I don't know where to start investigating here. What could cause the server to run out of disc space in one night?

Thanks, Matt

Also one of the weirdest things it seems. I reset everything yesterday eventing from an EC2 snapshot to a 200gb volume and attached it to the instance. Everything was working fine. I made some changes on the files, deleted some plugins, updated some settings.

And it seems this is all gone now. And I'm using an elastic IP, so I couldn't connect to a wrong device or something.

matt
  • 42,713
  • 103
  • 264
  • 397
  • 1
    check `df` command to find what disc is full. – Lamanus Oct 13 '19 at 10:28
  • @Lamanus see … https://i.imgur.com/fjZMj1C.png what does /dev/xvda1 100% use mean? – matt Oct 13 '19 at 15:31
  • try `df -h` I think the disc only has 20gb not 200gb. In my wordpress with small uploads, it uses 25gb and so your case is reasonable for the lack of spaces. – Lamanus Oct 13 '19 at 15:39
  • ok, but how so? My instance with t2.medium is called ```i-07a5...```. Under Volumes I see ```vol-0593…``` which is 200gb and is attached to ```i-07a5...```. It was created from a 20gb snapshot though. Do I have to "update" something here in order for the volume to then recognize that it should actually be 200gb? – matt Oct 13 '19 at 15:51
  • That is right. Usually it is correctly recognized when you restart your ec2. I was seen some command to do that without restart ec2 but I forgot. – Lamanus Oct 13 '19 at 15:55
  • check this. https://stackoverflow.com/questions/11014584/ec2-cant-resize-volume-after-increasing-size?r=SearchResults – Lamanus Oct 13 '19 at 15:57
  • It could be an issue on the underlying host that is running the VM. Perhaps try launch a new vm with the same code, but in a different region and see if you get the same issue? Only other thing I can think of is that you are writing a huge amount to the logs and that you need to rotate the logs. – OpenBSDNinja Oct 14 '19 at 08:23
  • @matt Will you please accept the second answer to your question here: https://stackoverflow.com/questions/3157372/css-horizontal-centering-of-a-fixed-div – Sid Nov 08 '19 at 23:21
  • @matt the old answer is outdated. the new one is appropriate for 2019. this affects a lot of people – Sid Nov 08 '19 at 23:22
  • Isn't this a symptom of running out of inodes? What does `df -i` say? – pkoch Jan 13 '20 at 10:57

3 Answers3

1

Bitnami Engineer here, you will probably need to resize the disk of your instance. But you can investigate those issues later, these commands will show the directories with large number of files 

cd /opt/bitnami
sudo find . -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
du -h -d 1

If MySQL is the service that is taking more space, you can try adding this line under the [mysqld] block of the /opt/bitnami/mysql/my.cnf configuration file

expire_logs_days = 7

That will force MySQL to purge the old logs of the server after 7 days. You will need to restart MySQL after that:

sudo /opt/bitnami/ctlscript.sh restart mysql

More information here:

https://community.bitnami.com/t/something-taking-up-space-and-growing/64532/7

Jota Martos
  • 4,548
  • 3
  • 12
  • 20
  • Hey thanks for sharing the info! I followed https://docs.bitnami.com/aws/how-to/modify-ebs-volume/ to expand my column and waited the AWS optimization of the column to finish, I then restarted my instance, but I still seeing the same "no space left" error, Bitnami says "The Bitnami initialization script will automatically resize the partition to use the new volume size when the instance restarts." but seems like not the case for me? – Neekey Apr 16 '20 at 14:03
0

What you need to do is increase the size of partition on the disk and the size of file system on that partition. Even you increased the volume size, these figure kept unchanged. Create another from snapshot would not help too. Check how to do it here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html

Nguyen Minh Chuong
  • 23
  • 3
0

Your df result shows 

Filesystem   1K—blocks       Used  Available Use% Mounted on 
udev           2014560          0    2014560   0% /dev 
tmpfs           404496       5872     398624   2% /run 
/dev/xvdal    20263528   20119048     128096 100% 
tmpfs          2022464          0    2022464   0% /dev/shm 
tmpfs             5120          0       5120   0% /run/lock 
tmpfs          2022464          0    2022464   0% /sys/fs/cgroup 
/dev/loop0       18432      18432          0 100% /snap/amazon—ssm—agent/1480 
/dev/loopl       91264      91264          0 100% /snap/core/7713 
/dev/loop2       12928      12928          0 100% /snap/amazon—ssm—agent/295 
/dev/loop3       91264      91264          0 100% /snap/core/7917 
tmpfs           404496          0     404496   0% /run/user/1000

where the root volume /deb/xvda1 has only 20GB and that is marked as 100% of the volume, not 200GB as you mentioned.

When you increase the volume size during the instance running, it is not automatically applied. In your EC2, you have to apply the change of volume as follows:

sudo resize2fs /dev/xvda1

and check the size of the volume by doing df -h then you will see the size is now 200GB.

Lamanus
  • 12,898
  • 4
  • 21
  • 47