mysqldump using a lot of space

Question

I have a server and the / partition is 20GB in size.

Databases are stored in /mnt/mysql-data partition is 500GB in size.

Now here's the problem. Whenever I run mysqldump it fills up / partition to 100%. I have already moved the tmpdir to /mnt/mysql-data/tmp. My databases are around 40GB all in-all now I want to back them up in /mnt/mysql-data/backups but I can't proceed because the / partition fills up to 100%. my mysqldump command is: mysqldump --all-databases > /mnt/s3share/backup.sql";

Server Details:

10.2.22-MariaDB-log MariaDB Server
CentOS Linux release 7.7.1908 (Core)

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G  217M  7.6G   3% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/xvda2       24G  2.4G   20G  11% /
/dev/xvda1      976M  168M  757M  19% /boot
tmpfs           1.6G     0  1.6G   0% /run/user/1000
/dev/xvdc1      500G  123G  378G  25% /mnt/mysql-data
tmpfs           1.6G     0  1.6G   0% /run/user/1001

MariaDB [db_inbox]> show global variables like "%tmp%";
+----------------------------+----------------------+  
| Variable_name              | Value                |  
+----------------------------+----------------------+  
| default_tmp_storage_engine |                      |  
| encrypt_tmp_disk_tables    | OFF                  |  
| encrypt_tmp_files          | OFF                  |  
| innodb_tmpdir              |                      |  
| max_tmp_tables             | 32                   |  
| slave_load_tmpdir          | /mnt/mysql-data/tmp  |  
| tmp_disk_table_size        | 18446744073709551615 |  
| tmp_memory_table_size      | 16777216             |  
| tmp_table_size             | 16777216             |  
| tmpdir                     | /mnt/mysql-data/tmp  |  
+----------------------------+----------------------+  
10 rows in set (0.00 sec)

update #1:

I forgot to mention that the *.sql backups are being written to /mnt/s3share/backups/ folder which is mounted as s3fs and its cache must be writing to /tmp and that may be a reason why / is being filled up while the sql dump is being created. However, when I run the backup and watch /tmp for changes, it doesn't show any growth. but when I do lsof command on /tmp I can see huge files being deleted. Could this be it?

@GerardH.Pille /var is in / partition. but the data is located in separate partition /mnt/mysql-data. there will be data growing in /var whenever mysqldump runs? — Christian Noel, Apr 20 '20 at 05:16
It sounds like the command is using `/tmp/`. Look for references to "tmp" in my.cnf. — Rick James, Apr 20 '20 at 05:36
@RickJames i already changed it before to go to ```/mnt/mysql-data/tmp``` would it still be using ```/tmp``` despite that? — Christian Noel, Apr 20 '20 at 05:54
/mnt/mysql-data/tmp wouldn't be a symbolic link, by any chance? — Gerard H. Pille, Apr 20 '20 at 11:38
@GerardH.Pille yes i constantly watch du -h on ```/``` partition and ```/var/log``` folder doesn't show any substantial growth. ```/mnt/mysql-data/tmp``` is not a symlink — Christian Noel, Apr 21 '20 at 03:59
Start the dump and when you see free space diminishing, run "find / -xdev -mmin -1 -ls" — Gerard H. Pille, Apr 21 '20 at 05:12
You didn't mention your OS. Your root is elastic, but it doesn't stretch? LOL — Gerard H. Pille, Apr 21 '20 at 05:27
@GerardH.Pille ah i updated a while ago that OS is Centos7. root isn't elastic but there's a mounted /mnt/s3share that uses amazon s3 / s3fs. — Christian Noel, Apr 21 '20 at 05:37
@GerardH.Pille i already found the problem, the fuse.s3fs was the one filling up /tmp all along and not mysqldump. already answered below. — Christian Noel, Apr 21 '20 at 07:00

score 1 · Answer 1 · answered Apr 20 '20 at 08:05

1

Look at your innodb_% variables and datadir. There could be something like a tablespace still somewhere else.

Failing that, look at the directory where the root bloat turns up using du while mysqldump is running.

answered Apr 20 '20 at 08:05

Gordan Bobić

971
4
11

Christian Noel · Accepted Answer · 2020-04-21T06:06:34.197

1

OK, I was able to solve this problem by moving the s3bucket cache dir from /tmp to /mnt/mysql-data/tmp.

Little did I know that fuse.s3fs was writing to /tmp and there's no way to track which file is growing using du -h /tmp

the code I was running was mysqldump --all-databases > /mnt/s3share/backup.sql where s3share is mounted using fuse.s3fs and has a cache directory targetting /tmp. this is why i thought mysqldump was causing all the growing storage use on root /.

after changing the cache dir of fuse.s3fs to /mnt/mysql-data/tmp, the problem was solved.

this was my mount command before /tmp:

datastore /mnt/s3share fuse _netdev,allow_other,use_cache=/tmp,passwd_file=$PASSWDFILE 0 0

then this is the new mount command /mnt/mysql-data/tmp:

datastore /mnt/s3share fuse _netdev,allow_other,use_cache=/mnt/mysql-data/tmp,passwd_file=$PASSWDFILE 0 0

edited Apr 21 '20 at 06:06

answered Apr 21 '20 at 05:51

Christian Noel

129
9

Strange how the /mnt/s3share is missing from the list of mounts in your original question. – Gerard H. Pille Apr 21 '20 at 08:25
yes i never thought it was going to be a main concern. i only realized recently that it could pose a big issue. – Christian Noel Apr 21 '20 at 08:30
i also added it under the update #1 – Christian Noel Apr 21 '20 at 08:31
1

That's what makes solving problems on StackExchange a challenge. Not the problem itself, it would have taken me a couple of minutes, but the contributers holding back information. – Gerard H. Pille Apr 21 '20 at 09:57
mysqldump -u$DB_USER --lockall-tables=false "$dbname"databases > "/mnt/mysql-datas3share/$dbnamebackup.sql" was originally mysqldump -u$DB_USER --lock-tables=false "$dbname" > "/mnt/mysql-data/$dbname.sql" – Gerard H. Pille Apr 21 '20 at 10:03
sorry about that. i was trying to limit information to the most basic i can and i thought it was mainly mysqldump that was causing an issue. – Christian Noel Apr 22 '20 at 00:27

score 0 · Answer 3 · answered Nov 29 '22 at 13:08

0

If you use s3fs 1.88 or later, it defaults to flushing the partial file to s3fs every 5 GB, controlled via -o max_dirty_data. This allows uploading files larger than temporary storage.

answered Nov 29 '22 at 13:08

Andrew Gaul

262
1
7

mysqldump using a lot of space

3 Answers3