1

I have a basic DigitalOcean droplet (1gb ram and 1 cpu) that has a few websites on it. There are a couple nodejs sites and a couple of Wordpress sites. All are pretty low traffic. It normally works just fine but today none of the wordpress sites were working. So I logged in and noticed that the cpu usage was pegged at 100% and it normally is at like 2-3%. I restarted the server and the cpu usage went down. But the sites were still not working. So when I tried to access the sites it errors and says there is not database connection. So I checked the status of mysql and it was not started. I tried to restart the service but that just fails. Here is the result of sudo systemctl status mysql

mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: activating (start) since Tue 2023-05-09 19:44:41 CDT; 48s ago
    Process: 3859 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
   Main PID: 3867 (mysqld)
     Status: "Server upgrade in progress"
      Tasks: 28 (limit: 1115)
     Memory: 491.4M
        CPU: 5.561s
     CGroup: /system.slice/mysql.service
             └─3867 /usr/sbin/mysqld

May 09 19:44:41 sk-webservice.com systemd[1]: mysql.service: Scheduled restart job, restart counter is at 7.
May 09 19:44:41 sk-webservice.com systemd[1]: Stopped MySQL Community Server.
May 09 19:44:41 sk-webservice.com systemd[1]: mysql.service: Consumed 5.529s CPU time.
May 09 19:44:41 sk-webservice.com systemd[1]: Starting MySQL Community Server...

And then I get the following in /var/log/mysql/error.log

2023-05-10T00:41:48.072177Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-0ubuntu0.22.04.1) starting as process 3037
2023-05-10T00:41:48.081239Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2023-05-10T00:41:48.660106Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-05-10T00:41:49.429920Z 4 [System] [MY-013381] [Server] Server upgrade from '80032' to '80033' started.
2023-05-10T00:42:44.456107Z 4 [ERROR] [MY-013178] [Server] Execution of server-side SQL statement '-- Deprecate spatial reference systems -- Deprecation must happen before new SRSs are added/updated since there -- typically are new SRSs with the same names. REPLACE INTO mysql.st_spatial_reference_systems(id, catalog_id, name, organization, organization_coordsys_id, definition, description) VALUES (6956, 1, 'VN-2000 / TM-3 zone 481 (6956 deprecated)', 'EPSG', 6956, 'PROJCS["VN-2000 / TM-3 zone 481",GEOGCS["VN-2000",DATUM["Vietnam 2000",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],TOWGS84[-191.90441429,-39.30318279,-111.45032835,0.00928836,-0.01975479,0.00427372,0.252906278],AUTHORITY["EPSG","6756"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.017453292519943278,AUTHORITY["EPSG","9122"]],AXIS["Lat",NORTH],AXIS["Lon",EAST],AUTHORITY["EPSG","4756"]],PROJECTION["Transverse Mercator",AUTHORITY["EPSG","9807"]],PARAMETER["Latitude of natural origin",0,AUTHORITY["EPSG","8801"]],PARAMETER["Longitude of natural origin",102,AUTHORITY["EPSG","8802"]],PARAMETER["Scale factor at natural origin",0.9999,AUTHORITY["EPSG","8805"]],PARAMETER["False easting",0,AUTHORITY["EPSG","8806"]],PARAMETER["False northing",500000,AUTHORITY["EPSG","8807"]],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["E",EAST],AXIS["N",NORTH],AUTHORITY["EPSG","6956"]]', 'Deprecated since EPSG Dataset 9.3. Superseded by SRID 5896.'); ' failed with error code = 1205, error message = 'Lock wait timeout exceeded; try restarting transaction'.
2023-05-10T00:42:44.463355Z 0 [ERROR] [MY-013380] [Server] Failed to upgrade server.
2023-05-10T00:42:44.463830Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-05-10T00:42:45.325713Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.04.1)  (Ubuntu).

So it seems like mysql was trying to upgrade and just failed? I don't remember running any upgrades recently. This is just a personal server and I am by no means a wiz on the server side of things. Does anyone know how I can fix this?

Steve K
  • 125
  • 1
  • 6
  • The error at the end of your long string is `Lock wait timeout exceeded; try restarting transaction`. That's unusual on its own upgrade process. I can't readily find anything about this elsewhere. You may need to restore a backup... – Halfgaar May 10 '23 at 01:14
  • 1
    Please add the output of `grep -i mysql var/log/apt/history.log` – shodanshok May 10 '23 at 05:38
  • @shodanshok I ended up just uninstalling and reinstalling mysql. I have no Idea what happened. I am about 100mb away from my ram limit most of the time so I wonder if maybe I ran out of ram and that caused some kind of issue and then when I rebooted something got corrupted. I have no idea. – Steve K May 10 '23 at 16:48
  • @SteveK Digital Ocean may have attempted to upgrade your instance - check with DO support team. – Wilson Hauck May 10 '23 at 20:23

1 Answers1

0

I got this same issue today and this is How I solved this.

  1. First of all I stopped mysql: service mysql stop

  2. After stopping mysql I ran MYSQLD with upgrade: /usr/sbin/mysqld --upgrade=minimal Make sure upgrade argument is minimal.

  3. Now if you get error like unable to create sock file then you can create folder like /var/run/mysqld

  4. Do check error logs in a seperate terminal while doing this: tail -f /var/log/mysql/error.log

  5. Now you will see mysqld running. Finally you can DUMP all the database by this cmd:

for DB in $(mysql -u root -pPassword -e 'show databases' -s --skip-column-names); do mysqldump -u root -pPassword $DB > "$DB.sql"; done

  1. Now you can uninstall mysql and install a stable version and Import all the databases.
K.K. Chauhan
  • 1