0

Our puppetmaster server was upgraded to the latest available on RHEL6 (puppet-2.7.25-2.el6.noarch). We are now receiving errors such as this on the master:

Aug 26 11:36:59 master1 puppet-master[5555]: Could not intern from yaml: instance of IO needed

And this on the client running 0.25.6 on RHEL4:

Aug 26 11:36:59 foo1 Puppet (err): Could not retrieve catalog from remote server: Error 400 on SERVER: Could not intern from yaml: instance of IO needed

The server is running using Apache and passenger which I'm not familiar with. Any ideas on how to troubleshoot this?

Update

Per @FelixFrank, after stopping httpd service, ran the following:

$ sudo puppet master --no-daemonize --verbose --trace

This provided me with the following traceback after a puppet client attempts to retrieve its catalog:

err: /usr/lib/ruby/1.8/yaml.rb:176:in `load'
/usr/lib/ruby/1.8/yaml.rb:176:in `parse'
/usr/lib/ruby/site_ruby/1.8/puppet/vendor/safe_yaml/lib/safe_yaml.rb:201:in `safe_load'
/usr/lib/ruby/site_ruby/1.8/puppet/vendor/safe_yaml/lib/safe_yaml.rb:141:in `load'
/usr/lib/ruby/site_ruby/1.8/puppet/network/formats.rb:6:in `intern'
/usr/lib/ruby/site_ruby/1.8/puppet/network/format_handler.rb:12:in `send'
/usr/lib/ruby/site_ruby/1.8/puppet/network/format_handler.rb:12:in `protect'
/usr/lib/ruby/site_ruby/1.8/puppet/network/format_handler.rb:26:in `intern'
/usr/lib/ruby/site_ruby/1.8/puppet/network/format_handler.rb:108:in `convert_from'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:25:in `extract_facts_from_request'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:38:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:195:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:111:in `do_find'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:24:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `call'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `synchronize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:126:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:141:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:124:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:202:in `main'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:146:in `run_command'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:416:in `hook'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:407:in `exit_on_fail'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:309:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:69:in `execute'
/usr/bin/puppet:4
err: Could not intern from yaml: instance of IO needed

Any ideas on what to try next?

Belmin Fernandez
  • 10,799
  • 27
  • 84
  • 148
  • Are both agent and master updated? - To debug this, please stop the master and start a debugging session using `puppet master --nodaemonize --verbose --trace`. The error should appear along with a call trace once the agent checks in. – Felix Frank Aug 26 '14 at 22:29
  • @FelixFrank how can I accomplish this when puppetmaster is running via Passenger? – Belmin Fernandez Sep 02 '14 at 12:26
  • 1
    Stop `apache`, then run the standalone master as described above. It should just work. Make sure it listens on the same port that your passenger vhost usually does. (Use `--masterport` if that's an issue.) – Felix Frank Sep 02 '14 at 15:10
  • That looks like the puppet from EPEL (which is ancient) rather than the one from puppetlabs (which is current). Do you specifically need to be this far out of date? Puppet Labs doesn't support 2.7 anymore. – Michael Hampton Sep 02 '14 at 15:31
  • The concern is that we can't get an agent more current than 0.25.6 running on RHEL4. We have seen issues before when connecting those clients to 3.x puppet masters. – Belmin Fernandez Sep 02 '14 at 15:34
  • Good luck. [RHEL 4 is on extended life support](https://access.redhat.com/support/policy/updates/errata/) and it's past time for migration plans for such boxes. I think Puppet Enterprise has a puppet agent build for EL4, though, if you want to go down that road. – Michael Hampton Sep 02 '14 at 15:56
  • There have been quite a few ruby/puppet/rails CVE's regarding yaml vulnerabilities. I'm guessing that puppet 2.7 won't parse the older yaml due to the security fixes. – Robert Sep 02 '14 at 21:16

1 Answers1

0

According to the Puppet changelog, 2.7.22 included changes on how YAML was being processed:

A critical vulnerability was found in puppet wherein it was possible for the puppet master to take YAML from an untrusted client via the REST API. This YAML could be deserialized to construct an object containing arbitrary code.

This was due to a security vulnerability. To temporarily mitigate the issue, I downgraded to an earlier version from PuppetLabs' repo and locked the puppet-server package via yum versionlock.

Belmin Fernandez
  • 10,799
  • 27
  • 84
  • 148