Suddenly my MySQL server will not start.
running mysql in the cli will return
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
I was running a process that probably killed the database and/or put it under excessive load. The same process caused a 504 error on the live database (allegedly) and I ran the same process on my localhost when it seem to freeze. Eventually the script 'completed' (process was not actually successful). However after completing, my MySQL died and I am failing to resurrect it.
I am running a Vagrant on a Virtual Box under windows 10. In the past this has happen and I've had to rebuild my entire localhost environment. This causes sufficient down time for various reasons and loss of the database is problematic. I would like to know why this is happening and how to get it working again.
Just to make it clear the MySQL server is not running. There are not process to kill. Rebooting my machine and then setting the localhost environment does nothing to fix this. I get the same error.
I have now spent quite a few hours hunting the internet for a solution with no success. Reviewed all the MySQL config files and they look fine. Tried restarting the MySQL server in various ways. Kill processes. Created a mysqld folder with correct permission and tonnes of other proposed solutions. I have not re-installed my mysql server as I do not want to lost the database.
$ systemctl status mysql.service
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2021-02-03 12:58:32 UTC; 26min ago
Process: 2752 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid (code=exited, status=1/FAILURE)
Process: 2730 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Feb 03 12:58:32 homestead systemd[1]: mysql.service: Failed with result 'exit-code'.
Feb 03 12:58:32 homestead systemd[1]: Failed to start MySQL Community Server.
Feb 03 12:58:32 homestead systemd[1]: mysql.service: Service hold-off time over, scheduling restart.
Feb 03 12:58:32 homestead systemd[1]: mysql.service: Scheduled restart job, restart counter is at 5.
Feb 03 12:58:32 homestead systemd[1]: Stopped MySQL Community Server.
Feb 03 12:58:32 homestead systemd[1]: mysql.service: Start request repeated too quickly.
Feb 03 12:58:32 homestead systemd[1]: mysql.service: Failed with result 'exit-code'.
Feb 03 12:58:32 homestead systemd[1]: Failed to start MySQL Community Server.
$ journalctl -xe
--
-- The start-up result is RESULT.
Feb 03 13:07:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:08:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:09:00 homestead systemd[1]: Starting Clean php session files...
-- Subject: Unit phpsessionclean.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit phpsessionclean.service has begun starting up.
Feb 03 13:09:01 homestead CRON[2980]: pam_unix(cron:session): session opened for user root by (uid=0)
Feb 03 13:09:01 homestead CRON[2981]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb 03 13:09:01 homestead CRON[2980]: pam_unix(cron:session): session closed for user root
Feb 03 13:09:01 homestead systemd[1]: Started Clean php session files.
-- Subject: Unit phpsessionclean.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit phpsessionclean.service has finished starting up.
--
-- The start-up result is RESULT.
Feb 03 13:09:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:10:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:11:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:12:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:13:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:14:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:15:01 homestead CRON[3048]: pam_unix(cron:session): session opened for user root by (uid=0)
Feb 03 13:15:01 homestead CRON[3049]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Feb 03 13:15:01 homestead CRON[3048]: pam_unix(cron:session): session closed for user root
Feb 03 13:15:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:16:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:17:01 homestead CRON[3053]: pam_unix(cron:session): session opened for user root by (uid=0)
Feb 03 13:17:01 homestead CRON[3054]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 03 13:17:01 homestead CRON[3053]: pam_unix(cron:session): session closed for user root
Feb 03 13:17:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:18:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:19:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:20:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:21:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:22:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:23:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:24:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
Feb 03 13:25:01 homestead CRON[3067]: pam_unix(cron:session): session opened for user root by (uid=0)
Feb 03 13:25:01 homestead CRON[3068]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Feb 03 13:25:01 homestead CRON[3067]: pam_unix(cron:session): session closed for user root
Feb 03 13:25:07 homestead env[845]: [APIv1] KEEPALIVE /api/v1/events
var/logs/mysql/error.log
2021-02-04T04:50:22.916611Z 0 [Note] Binlog end
2021-02-04T04:50:22.916664Z 0 [Note] Shutting down plugin 'CSV'
2021-02-04T04:50:22.916939Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
2021-02-04T04:50:23.442484Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-02-04T04:50:23.443904Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.27-0ubuntu0.18.04.1) starting as process 2399 ...
2021-02-04T04:50:23.449165Z 0 [Note] InnoDB: PUNCH HOLE support available
2021-02-04T04:50:23.449188Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-02-04T04:50:23.449195Z 0 [Note] InnoDB: Uses event mutexes
2021-02-04T04:50:23.449201Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2021-02-04T04:50:23.449206Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-02-04T04:50:23.449214Z 0 [Note] InnoDB: Using Linux native AIO
2021-02-04T04:50:23.449505Z 0 [Note] InnoDB: Number of pools: 1
2021-02-04T04:50:23.449607Z 0 [Note] InnoDB: Using CPU crc32 instructions
2021-02-04T04:50:23.452188Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-02-04T04:50:23.464668Z 0 [Note] InnoDB: Completed initialization of buffer pool
2021-02-04T04:50:23.466372Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-02-04T04:50:23.477777Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2021-02-04T04:50:23.478809Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 101344176183
2021-02-04T04:50:23.478854Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 101344232648
2021-02-04T04:50:23.487359Z 0 [Note] InnoDB: Database was not shutdown normally!
2021-02-04T04:50:23.487371Z 0 [Note] InnoDB: Starting crash recovery.
2021-02-04T04:50:23.501921Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
2021-02-04T04:50:23.512026Z 0 [Note] InnoDB: Apply batch completed
2021-02-04T04:50:23.639660Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-02-04T04:50:23.639718Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-02-04T04:50:23.646302Z 0 [Warning] InnoDB: 1048576 bytes should have been written. Only 1032192 bytes written. Retrying for the remaining bytes.
2021-02-04T04:50:23.646330Z 0 [Warning] InnoDB: Retry attempts for writing partial data failed.
2021-02-04T04:50:23.646339Z 0 [ERROR] InnoDB: Write to file ./ibtmp1failed at offset 11534336, 1048576 bytes should have been written, only 1032192 were written. Operating system error number 28. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.
2021-02-04T04:50:23.646348Z 0 [ERROR] InnoDB: Error number 28 means 'No space left on device'
2021-02-04T04:50:23.646354Z 0 [Note] InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html
2021-02-04T04:50:23.646360Z 0 [ERROR] InnoDB: Could not set the file size of './ibtmp1'. Probably out of disk space
2021-02-04T04:50:23.646366Z 0 [ERROR] InnoDB: Unable to create the shared innodb_temporary
2021-02-04T04:50:23.646371Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2021-02-04T04:50:24.149465Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2021-02-04T04:50:24.149490Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2021-02-04T04:50:24.149496Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2021-02-04T04:50:24.149502Z 0 [ERROR] Failed to initialize builtin plugins.
2021-02-04T04:50:24.149507Z 0 [ERROR] Aborting
Does any one have any ideas? I'm really stuck. :(
