2

I'm trying to run Fedora Cloud Base 36 on an e2-micro in the US central region of Google Cloud Platform (GCP). I haven't upgraded packages in a couple months so I tried to do that today. When I run:

$ sudo dnf upgrade

the OS runs out of memory, kills my command, and terminates my SSH session. From journalctl logs:

sudo[917]:  jrefior : TTY=pts/0 ; PWD=/home/jrefior ; USER=root ; COMMAND=/usr/bin/dnf upgrade
sudo[917]: pam_unix(sudo:session): session opened for user root(uid=0) by jrefior(uid=1000)
systemd-oomd[613]: Killed /user.slice/user-1000.slice/session-3.scope due to memory used (955539456) / total (1007927296) and swap used (945311744) / total (1007>
systemd-oomd[613]: Killed /user.slice/user-1000.slice/session-3.scope due to memory used (956289024) / total (1007927296) and swap used (981708800) / total (1007>
systemd[1]: session-3.scope: systemd-oomd killed 6 process(es) in this unit.
systemd-logind[798]: Session 3 logged out. Waiting for processes to exit.
systemd[1]: session-3.scope: Deactivated successfully.
systemd[1]: session-3.scope: Consumed 10.737s CPU time.
systemd-logind[798]: Removed session 3.

An e2-micro has 1 GB memory, and the instance was created with 1 GB swap as well:

$ free -m
            total       used        free    shared  buff/cache   available
Mem:            961         146     558         0       256         680
Swap:           960         27      933

Can Fedora Cloud Base be managed on 2 GB of memory (including swap)? GCP e2-micro specs are here. Is there a way to upgrade kernel and packages without using so much memory?

Would I run into a problem upgrading Fedora Cloud Base as well when the time comes?

jrefior
  • 123
  • 5
  • 1
    To decrease memory usage, I preferred to [manually upgrade](https://docs.fedoraproject.org/en-US/fedora/latest/system-administrators-guide/kernel-module-driver-configuration/Manually_Upgrading_the_Kernel/) the kernel. If this works, try this. – Jeffrey D. Jan 16 '23 at 22:18
  • Thanks @JeffreyD., I hadn't seen that doc before – jrefior Jan 17 '23 at 03:11
  • 1
    Can you try upgrading your VM instance's specifications first, and if that succeeds, the kernel and packages? Try to restore your VM's original specifications. Make a security snapshot first before executing that. – Jeffrey D. Jan 17 '23 at 17:11
  • 1
    Thanks @JeffreyD., that's a good idea. After reading John Mahowald's answer and other information, I'm considering switching to Arch for my e2-micro. Seems like a better fit to the resources available. – jrefior Jan 18 '23 at 22:47

2 Answers2

2

No, 1 GB of RAM is not enough for package management of a Fedora based system. Which in my opinion includes RHEL and rebuilds, but the extent of the problem on the size of the repos involved. Yes, a distro major version upgrade transaction is among the biggest, and could be problematic as well.

Paging out during a package transaction is a miserable experience. Slow. And even with the swap possibly oom killed in a dangerous place. I am being unfair to swap space by only calling this 1 GB, but the memory system behaves differently than if you had 2 GB of DRAM.

For recent reports of this happening, see mailing list thread dnf makecache memory usage increase, and Bugzilla "dnf update" runs out of memory on swapless machines with 1G or less of RAM. Note that the entire dnf metadata is being converted into some libsolv database in memory, on top of what it does to parse those dependencies.

If you moved your instances to image based updates, that could be done once on a bigger box. Upgrade a primary copy, make a disk image out of that, and boot new instances from the updated one. Likely a significant change of how you do updates. See also: rpm-ostree.

Commenters on the bug have noted file lists blowing up the size of metadata. Especially with the number of packages in Fedora, how often they update, and how many have still have legacy file dependencies. Consider maintaining your own custom repos of just a subset of the packages you use.

microdnf is the package manger without Python. While there might be some memory savings to switching, its authors have not laid out in detail its limitations. Possibly wait on this one, Fedora is inevitably going to change the package manager again anyway.

John Mahowald
  • 32,050
  • 2
  • 19
  • 34
0

You can create a swapfile and utilize it while running dnf updates.

$ sudo dd if=/dev/zero of=/swapfile count=1024 bs=1MiB
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ sudo dnf update

Once the updates are finished you can turn swap off by running:

$ sudo swapoff

When using a swapfile over the long-term there were spurious instance stability issues, but these were mainly over come by using zram. Unfortunately even with 384MB of zram enabled updates would be killed by oom, otherwise the system was stable and has run a LAMP stack for over a year.

Jamie Fargen
  • 1
  • 1