0

I seem to be having memory problems in my t2.micro instance (1GB) running nginx, mariadb, php and WordPress.

I can see that mariadb.service is being killed regularly (I have used grep -e kill /var/log/messages sample output below). As you can see mysqld is killing mariadb (isn't this suicide?).

I have tried various tweaks and tuning for mariadb but I think it is a more an overall system issue.

I haven't been able to isolate when the "crashes" occur. I can be happily logged into WordPress admin, working away, and when I switch to a new area (prompting a dbase call) the site hangs. SSH connections also hang at the same time.

Is the t2.micro just not powerful enough?

Apr  9 11:22:32 ip-172-31-20-68 systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Apr  9 11:23:25 ip-172-31-20-68 kernel: mysqld invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Apr  9 11:23:25 ip-172-31-20-68 kernel: oom_kill_process.cold.28+0xb/0x10
Apr  9 11:23:25 ip-172-31-20-68 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=6383,uid=27
Apr  9 11:23:25 ip-172-31-20-68 systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Apr  9 11:27:04 ip-172-31-20-68 kernel: tuned invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Apr  9 11:27:04 ip-172-31-20-68 kernel: oom_kill_process.cold.28+0xb/0x10
Apr  9 11:27:04 ip-172-31-20-68 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=6483,uid=27
Apr  9 11:27:51 ip-172-31-20-68 NetworkManager[928]: <info>  [1617967671.1410] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
Apr  9 11:27:51 ip-172-31-20-68 NetworkManager[928]: <info>  [1617967671.1411] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
Apr  9 12:04:48 ip-172-31-20-68 kernel: mysqld invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Apr  9 12:04:48 ip-172-31-20-68 kernel: oom_kill_process.cold.28+0xb/0x10
Apr  9 12:04:48 ip-172-31-20-68 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=1746,uid=27
Apr  9 12:04:48 ip-172-31-20-68 systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Apr  9 12:06:04 ip-172-31-20-68 kernel: tuned invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Apr  9 12:06:04 ip-172-31-20-68 kernel: oom_kill_process.cold.28+0xb/0x10
Apr  9 12:06:04 ip-172-31-20-68 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-3.scope,task=dnf,pid=1982,uid=0
Apr  9 12:08:30 ip-172-31-20-68 kernel: php-fpm invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Apr  9 12:08:31 ip-172-31-20-68 kernel: oom_kill_process.cold.28+0xb/0x10
Apr  9 12:08:31 ip-172-31-20-68 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb.service,task=mysqld,pid=2126,uid=27
Apr  9 12:08:31 ip-172-31-20-68 systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
dtw
  • 103
  • 3

1 Answers1

2

The OOM killer is a kernel function that operates on behalf of the process currently allocating memory. This can be the same process that is chosen to be killed to free memory.

The selection rules generally choose the process with the largest mapping of anonymous memory (i.e. discounting memory mapped files) to physical pages (i.e. discounting memory that is swapped out or has never been written to).

It is usually possible to tune databases to use less memory at the expense of performance, and I'd also expect the default tuning to assume that the database server is a dedicated machine and should devote all of its resources to this single task, which is not what you are doing here.

So there are multiple avenues here:

  1. run a separate VM that is dedicated to the database
  2. tune database settings to use less memory
  3. use a larger VM

Databases are usually good at adapting to however much memory you have available and use it for caches that are known to the query optimizer and can be part of cost calculation (as opposed to OS-level disk caches, which are unpredictable), so either limiting that to some value that leaves enough space for the other software, or making sure no other software needs space solves that nicely.

If that is still insufficient, you will need a larger VM.

Adding swap space is not helpful, because it makes the access times of the database internal caches unpredictable, so the database will start creating suboptimal query plans as it assumes that data is cached and an access to the in-memory copy will be faster than reloading the data from disk -- but swap space on a VM where you pay for I/O is not a good idea either way.

Simon Richter
  • 3,317
  • 19
  • 19