0

I have a VPS running for a long time. It's for some small websites and for private mail for myself and family. According to directadmin I was only using 10 gigabyte of space and the VPS should have 50 gigabyte.

Last week I was testing SQL queries. Yesterday I got the first error when phpmyadmin could not perform one SQL due to: no space left. Small queries did work. I updated my VPS plan from 50gb to 150gb and did an update:

yum check-update
yum update -y
/usr/local/directadmin/custombuild/build update
/usr/local/directadmin/custombuild/build update_versions
reboot

After the reboot Apache, Exim, MariaDB and Named stopped. Directadmin is running so I can access it but all my websites and my email are now unreachable.

When I performed:

[root@server domains]# systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/etc/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: failed (Result: resources)

Jun 20 21:53:32 server.intermagic.nl systemd[1]: httpd.service failed to run 'start' task: No space left on device
Jun 20 21:53:32 server.***.nl systemd[1]: Failed to start The Apache HTTP Server.
Jun 20 21:53:32 server.***.nl systemd[1]: Unit httpd.service entered failed state.
Jun 20 21:53:32 server.***.nl systemd[1]: httpd.service failed.
Jun 20 21:57:43 server.***.nl systemd[1]: httpd.service failed to run 'start' task: No space left on device
Jun 20 21:57:43 server.***.nl systemd[1]: Failed to start The Apache HTTP Server.
Jun 20 21:57:43 server.***.nl systemd[1]: httpd.service failed.

After a lot of reboots I now get:

[admin@server tmp]$ systemctl status httpd.service
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/etc/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: failed (Result: resources)

What could be the cause of it? Really too much of bad SQL queries? And what can I do about it? I can re-install everything, but...

Edit 1:

Filesystem                          Size    Used    Avail  Use%   Mounted on 
devtmpfs                            1.9G    0       1.9G   0%     /dev
tmpfs                               1.9G    0       1.9G   0%     /dev/shm
tmpfs                               1.9G    89M     1.8G   5%     /run
tmpfs                               1.9G    0       1.9G   0%     /sys/fs/cgroup
/dev/mapper/vg_directadmin-lv_root  48G     20G     28G    42%    /
/dev/vda1                           497M    262M    236M   53%    /boot
/dev/mapper/vg_directadmin-lv_tmp   1014M   1014M   0      100%   /tmp
Mircea Vutcovici
  • 17,619
  • 4
  • 56
  • 83
MacQwerty
  • 1
  • 1
  • 1
    What does `df -h` say? – Esa Jokinen Jun 21 '20 at 10:32
  • Filesystem Size Used Avail Use% Mounted on devtmpfs 1.9G 0 1.9G 0% /dev tmpfs 1.9G 0 1.9G 0% /dev/shm tmpfs 1.9G 89M 1.8G 5% /run tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/mapper/vg_directadmin-lv_root 48G 20G 28G 42% / /dev/vda1 497M 262M 236M 53% /boot /dev/mapper/vg_directadmin-lv_tmp 1014M 1014M 0 100% /tmp Is it the /dev/mapper/vg_directadmin-lv_root ?? – MacQwerty Jun 21 '20 at 11:13
  • Could you edit your question and add the output of `pvs` and `lsblk` ? – Mircea Vutcovici Jun 21 '20 at 12:05

2 Answers2

2

From the df -h output it should be pretty easy to spot the Use 100% with Avail 0.

Your /tmp is full. Adding capacity to the / doesn't solve this.

The question on how to increase directadmin tmp is discussed on DirectAdmin forums.

Esa Jokinen
  • 46,944
  • 3
  • 83
  • 129
0

Check the system has enough free extents (PFree column) with:

sudo pvs

If you have enough free LVM extents you can expand the /tmp volume with 300MB by running:

sudo lvextend -L +300m --resizefs /dev/mapper/vg_directadmin-lv_tmp

If you don't have enough free extents:

  • You need to see on which /dev/vd[ab][0-9] block device you have /tmp. lsblk will help you identify this.
  • Expand the disk of the VM that is storing the /dev/vd[ab][0-9] block device. This can be done online in modern clouds and virtualization platforms.
  • Ask the VM to rescan the block device for size changes. Run as root user the following command, make sure you use the right vd... device: echo 1 > /sys/block/vd..../device/rescan
  • Expand the LVM physical disk: pvresize /dev/vd....
  • Expand the LV and the filesystem: lvextend -L +300m --resizefs /dev/mapper/vg_directadmin-lv_tmp
  • no reboot should be needed

You can also use this script as root and it will generate the command to expand the /tmp volume: sudo bash vol_resize.sh --block-device /dev/mapper/vg_directadmin-lv_tmp

Mircea Vutcovici
  • 17,619
  • 4
  • 56
  • 83
  • I got my server running after Esa his reply. I will look into your answer because I suspect the problem will re-occur. When I now check the sudo pvs command (after removing tmp and a reboot) I have pfree=0 and Psize <49.51g – MacQwerty Jun 21 '20 at 17:33