0

MySQL version: mysql Ver 8.0.25-0ubuntu0.20.04.1 for Linux on x86_64 ((Ubuntu))

I have a server that hosts a MySQL database and Apache Airflow. It was working fine for a couple of months. For a couple of days, my services are constantly crashing.

I see that the error is due to insufficient Disk Space: These are the errors that I see when I try to run my Airflow Service.

enter image description here

enter image description here

This is happening even though I see there is disk space left: enter image description here

I figured out that this could be due to a MySQL bug, https://community.oracle.com/tech/developers/discussion/4267115/mysql-daemon-not-releasing-deleted-temp-file

But I have a newer version of MySQL installed so this should not happen.

Tried restarting the server a couple of times.

Tried killing the MySQL services manually, but it always goes back to the same state.

So I am not even able to get it fixed temporarily.

enter image description here

Not able to understand how to fix the error permanently, any help will be appreciated.

ofnowhere
  • 101
  • 1
  • Does this answer your question? [Disk full, du tells different. How to further investigate?](https://serverfault.com/questions/275206/disk-full-du-tells-different-how-to-further-investigate) – Michael Hampton Jun 21 '21 at 20:40
  • Lower `max_connections` in my.cnf to only 30. Is `tmpdir` being set in the config file? – Rick James Jun 22 '21 at 05:53
  • @MichaelHampton no I don't think that is realted. – ofnowhere Jun 22 '21 at 14:30
  • @RickJames Lowered max connections, still the same. – ofnowhere Jun 22 '21 at 14:30
  • Did you even look? What were the inode counts? – Michael Hampton Jun 22 '21 at 14:31
  • Yes, @MichaelHampton, Inodes were 100% used, I increased the disk space. So seems it got solved temporarily. Mysql is still accessing those deleted files. So I meant, MySQL accessing deleted temp files is unrelated. – ofnowhere Jun 22 '21 at 15:05
  • I think you'll just have to restart MySQL. – Michael Hampton Jun 22 '21 at 15:08
  • I did that a couple of times, the PID changes, but it keeps accessing those files. – ofnowhere Jun 22 '21 at 15:10
  • 100% inodes used -- sounds like lots of tiny files on the file system. In my several decades of using *nix, I don't remember running out of inodes before running out of space. I wouldn't even know what OS parameter to change to increase the ratio between inode and data blocks. – Rick James Jun 22 '21 at 15:45
  • @RickJames 100% inodes were because of Airflow, Each dag execution in Airflow creates a log directory(multiple jobs are running every couple of minutes), we have logrorate to delete the log files, but directories remain here. I suppose that is what caused it. – ofnowhere Jun 22 '21 at 19:03
  • 1
    I will write a cron to clean up empty directories and then try that. – ofnowhere Jun 22 '21 at 19:03
  • @ofnowhere - Suggest filing a bug report with Airflow, – Rick James Jun 22 '21 at 19:05
  • Not sure if it is a bad, but a bad implementation of a feature. – ofnowhere Jun 23 '21 at 16:57

1 Answers1

1

Out of inodes? Check with df --inodes

A quick, but temporary, fix would be to toss files in .Trash and Download

Maybe there are lots of unnecessary empty files: https://www.interserver.net/tips/kb/solution-running-inodes/ https://askubuntu.com/questions/231585/running-out-of-inodes

Maybe "compressing" a few little-used disk trees would help. (You will need a few free inodes before trying this.) Or tar up some trees that you don't need.

(etc)

Rick James
  • 2,463
  • 1
  • 6
  • 13