We have a puppet server running 3.8.7 on RHEL 6.10. It experienced an OOM event this morning then the server recovered without intervention. While looking into this I noticed the server became very slow to respond. I saw that it suddenly had over 3200 processes running and was about to crash. I was able to reboot the box and it came back up fine. Everything now seems normal except for the fact that there are about 15 files now in /var/lib/pgsql/data/base// that are eating up 1G each. These are all timestamped with todays date and were not there until after the reboot. Does anyone know if these files are safe to delete? I am not familar with this particular folder or how it interacts with puppetmaster.
Asked
Active
Viewed 245 times
1
-
try to check if you are using puppetdb, puppet doesn't uses database by default – c4f4t0r Feb 16 '19 at 16:24
-
No, I only see puppet and puppetmaster installed. No puppetdb. – user53029 Feb 16 '19 at 16:27
-
1try to get inside the psql shell and check the databases content before delete something – c4f4t0r Feb 16 '19 at 16:30
-
Confirmed there is legit data inside the db. – user53029 Feb 16 '19 at 16:49
-
It's likely the older storeconfigs if there's no PuppetDB. (Look for `storeconfig` in `puppet.conf`.) It's used for exported resources and you may have found a bug if it blew out 3200 processes and OOM'd. The good news is, the current version of Puppet is 6.2, so maybe it's been fixed. :) – Aaron Copley Feb 18 '19 at 01:59
-
I'll take a look, thanks. Its really strange the OOM's happen everyday at the same exact time. – user53029 Feb 18 '19 at 20:57