Puppet hang on 100% CPU usage

Question

We're running puppet to manage our AWS instances and are migrating from a custom apt module for it, to the puppetlabs hosted apt module. Everything seems to work fine until during an puppet apply session, the puppet process will hang on %100 for as long as we let it.

I believe I've narrowed it down to puppet hanging while waiting for some "response" in the apt module, but I can't figure it out any further. I see a few related bugs and issues (like here http://projects.puppetlabs.com/issues/12185#note-27), but I can't find a good answer. Any help would be appreciated.

We're running puppet 2.7.20 on ubuntu 10.10. Puppet apt module version is 1.1.0.

As requested, here are the last few debug lines before the hang:

    debug: /Stage[main]/Kpython/Kpython::Pip_install[simplejson]/Exec[kpython_pip_install         simplejson]: Autorequiring File[/mnt/tmp]
    debug: /Stage[puppet1]/Kbase::Users/Kbase::Users::Account[tfall]/File[/home/tfall/.ssh]: Autorequiring Group[users]
    debug: /Stage[puppet1]/Kbase::Users/Kbase::Users::Account[tfall]/File[/home/tfall/.ssh]: Autorequiring User[tfall]
    debug: /Stage[puppet1]/Kbase::Filesystem/File[/usr/local/krux/kbase]: Autorequiring File[/usr/local/krux]
    debug: /Stage[main]/Kvagrant/Kruby::Setup_gemset[kvagrant]/Kruby::Wrapper[kvagrant-irb]/Kutil::Symlink_bin[kvagrant-irb]/File[/usr/local/bin/kvagrant-irb]: Autorequiring File[/usr/local/bin]
    debug: /Stage[main]/Kvagrant/Kruby::Setup_gemset[kvagrant]/Kruby::Wrapper[kvagrant-irb]/Kutil::Symlink_bin[kvagrant-irb]/File[/usr/local/bin/kvagrant-irb]: Autorequiring File[/usr/local/krux/components/kruby/bin/kvagrant-irb]
    debug: /Stage[main]/Kpython/Kpython::Pip_install[boto]/Exec[kpython_pip_install boto]: Autorequiring File[/mnt/tmp]
    debug: /Stage[script1]/Kbase::Users::Finalize/Kbase::Users::Init_user[dbrans]/File[/home/dbrans/.ssh/known_hosts]: Autorequiring User[dbrans]
    debug: /Stage[puppet1]/Kbase::Users/Kbase::Users::Account[vsvaidya]/File[/home/vsvaidya]: Autorequiring Group[users]
    debug: /Stage[main]/Kvagrant/File[/usr/local/krux/components/kvagrant]: Autorequiring File[/usr/local/krux/components]
    debug: /Stage[main]/Kvagrant/File[/usr/local/krux/components/kvagrant]: Autorequiring Group[vagrant]
    debug: /Stage[main]/Kvagrant/File[/usr/local/krux/components/kvagrant]: Autorequiring User[vagrant]

The next steps in this puppet run level should be apt module related.

This is the output of strace (with both ruby 1.8.7 and ruby 1.9.1):

         rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
         rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
         rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
         rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
         rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
         rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
         rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
         rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
         rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0

Can you provide the output when `--debug` is added to the run? — Shane Madden, Jun 11 '13 at 00:59
Sure, although that would be an extremely long output, with nothing unusual before it just stops. — user177334, Jun 11 '13 at 01:04
Agreed. However, I've been unable to do this. My thinking is that it's waiting for a response, but I can't find out from what. — user177334, Jun 11 '13 at 01:25
Do you have any files recursing a directory? http://serverfault.com/questions/324757/puppet-agent-using-100-cpu-on-a-single-node-for-an-hour. You might try running puppet under strace. — Zoredache, Jun 11 '13 at 01:25
Checked for recursion, but couldn't find any in our user.pp setup. — user177334, Jun 11 '13 at 01:29
And couldn't find any recursion in files that apt should be dealing with (outside of the apt module manifests themselves) — user177334, Jun 11 '13 at 01:36
Try to use `strace` as @Zoredache said. Complementary, you can use `pstack` and `ltrace` too. — Mircea Vutcovici, Jun 11 '13 at 03:08
I did run strace early on, and I've added it to the question. I know it has something to do with ruby, but I wasn't sure it was the issue. I also just tried ruby 1.9.1 with the same result. — user177334, Jun 11 '13 at 05:40
Make sure you're running `strace` with the `-f` flag so it follows forked processes. — jgoldschrafe, Jun 13 '13 at 05:00

score 0 · Answer 1 · answered Jun 11 '13 at 02:59

0

What ruby are you running? This feels something like this issue, which says it's FreeBSD but turned out to be a ruby issue.

answered Jun 11 '13 at 02:59

Bill Weiss

10,979
3
38
66

Originally run with 1.8.7 but I've also tried it with ruby 1.9.1. – user177334 Jun 11 '13 at 05:42

Puppet hang on 100% CPU usage

1 Answers1