How can I move /var/log directory

Question

On our production server there is a small drive for the root mount point /, /var/log is taking too much space and I have to manually delete some files. How can I move /var/log/ to let's say /home/log WITHOUT REBOOTING?

Here is the thing I thought:

$ mkdir /home/log
$ rsync -a /var/log /home/log
$ mount --bind /home/log /var/log
$ /etc/init.d/rsyslog restart

But I know that some services use file descriptors, so they'll continue to use /var/log or inodes.

Answer 1

Proper design

I assume you are unable to simply extend the filesystem in question (using lvextend && ext2online), because you do not use LVM or use wrong filesystem type.

Your approach

What you've proposed might work if you signal the daemons with SIGHUP (kill -1 pid). Obviously you would need to later on "mount -o bind / /somewhere" and clean up what has been left underneath mounted /var/log. But it has a bad smell for me, especially for production.

Avoid downtime, have a clean result (but complicated to do)

Forget about "mount -o bind" idea, create a new LV/partition, but don't mount it yet.

lsof | grep /var/log             # lists open files in /var/log

For each daemon that has any open file (I would expect at least syslog, inetd, sshd):

• reconfigure the daemon no to log to /var/log
• refresh the daemon (kill -1 or /etc/init.d/script reload)
• confirm with lsof | grep /var/log that daemon has closed its files

Mount over /var/log. Restore old configurations, SIGHUP/reload daemons again.

Easy way (downtime)

Create a new LV/partition and mount it properly over either /var or /var/log. The easy way is to take down the server to maintenance mode (single-user mode), and use the actual console (not ssh) for the operation.

Answer 2

Everyone else's answers are excellent and correct, and you should definitely read them first.

I just thought I'd share this because it makes for easy copy-and-paste, if your case turns out to be quite a simple one like mine was:

Stop the syslog and copy current logs out:

service rsyslog stop
mkdir -p /tmp/varlog
cp -r /var/log/* /tmp/varlog

then, mount your new location over /var/log. Say it's a new device called /dev/sdb

mount /dev/sdb /var/log

now you can copy files back and restart the syslog:

cp -r /tmp/varlog/* /var/log
rm -rf /tmp/varlog
service rsyslog start

Assuming this all happens quite early on in the life of your machine, rsyslog is likely to be the only daemon running. YMMV!

PS - you'll be wanting to add it to your fstab as well probably. Here's one way of doing that, again assuming a very straightforward mount:

cat /etc/mtab |grep /var/log >>/etc/fstab

(cf https://serverfault.com/a/267610/80606 about catting mtab to fstab)

Answer 3

Another thing that you could do is:

• Stop the processes that have open files on /var/log
• Verify that there aren't any processes with open files on /var/log (using lsof as kubanskamac suggested)
• Move your /var/log to another partition with enough free space (following your example, that would be /home/log)
• Create a symbolic link from /var/log to /home/log (ln -s /home/log /var/log)
• Restart the processes that you stopped in the first step

Please note that this is far from what I'd consider as a good practice. It's just a workaround so that you don't have to shutdown the server. The right solution would be to create a new /var or /var/log partition with enough space (or expand the current one),

Answer 4

To move the log directory to a different location:

sudo service rsyslog stop
cd /var
sudo mv log /path/to/new/location/
sudo ln -s /path/to/new/location/log
sudo service rsyslog start

Answer 5

Another solution based on @hwjp , if you can't use another drive volume to move them, you can create a virtual drive volume into another volume who has more space available (that is my case) :

Create virtual volume:

A) do : sudo dd if=/dev/zero of=VHD-log.img bs=1M count=1200

• if=/dev/zero: input file to provide a character stream for initialising data storage
• of=VHD.img: image file to be created as storage volume on /thevolumeofyourchoice
• bs=1M: read and write up to 1M at a time
• count=1200: copies only 1200M (1GB) input blocks

B) do : sudo mkfs -t ext4 /thevolumeofyourchoice/VHD-log.img Format the EXT4 file system type in the VHD-log image file with the mkfs utility.

C) do : sudo mkdir /thevolumeofyourchoice/vlog Mount VHD-log to a directory (mount point)

D) do : sudo mount -t auto -o loop /thevolumeofyourchoice/VHD-log.img /thevolumeofyourchoice/vlog

D1) To mount VHD-log at system boot to the final directory, add this entry in the /etc/fstab file.

/thevolumeofyourchoice/VHD-log.img  /var/log/  ext4    defaults        0  0

Move old log files:

E) do :

• service rsyslog stop
• lsof | grep /var/log to lists open files in /var/log and switch off needed daemon (apach2,freshclam in my case)
• cp -rp /var/log/* /thevolumeofyourchoice/vlog (cp -p same as --preserve=mode,ownership,timestamps)

F) do :

• sudo umount /thevolumeofyourchoice/vlog
• sudo mv /var/log /var/log-old
• sudo mkdir /var/log
• sudo chgrp syslog /var/log
• sudo mount -t auto -o loop /thevolumeofyourchoice/VHD-log.img /var/log

G) do : service rsyslog start and restart others services you stopped

Finally double check :

You ca do a - lsof | grep /var/log to lists open files in /var/log and verify they point to /var/log and not /var/log-old

You can mv, backup or delete /var/log-old after all is ok.

Answer 6

I would just:

apt install lsyncd
mv /var/log /varlog.root
mkdir /var/log /home/log
mount --bind /home/log /var/log

# set up lsyncd appropriately to sync real-time between /varlog.root & /var/log
service lsyncd start
sleep 3m

service rsyslog restart
# Proceed with all the other daemons
# to switch them over to the new /var/log and release /varlog.root
sleep 3m

# Then do this a few times until no processes use the old dir
lsof +D /varlog.root && rm -rf /varlog.root