Chef-client throws 403 forbidden intermittently

Question

I am using a brand new chef-server (not enterprise), and running chef-client throws a 403 forbidden error approximately every 1 out of 5 runs. Anyone know why this might occur?

chef-server 11.0.12 via getchef.com/chef/install
chef-client 11.12.4 via rubygem
ruby 2.1.2
Debian Wheezy 7.5

The error:

198.xx.xxx.xxx ================================================================================
198.xx.xxx.xxx Error executing action `create` on resource 'template[/var/www/mysite.com/shared/config/database.yml]'
198.xx.xxx.xxx ================================================================================
198.xx.xxx.xxx
198.xx.xxx.xxx Net::HTTPServerException
198.xx.xxx.xxx ------------------------
198.xx.xxx.xxx
198.xx.xxx.xxx 403 "Forbidden"
198.xx.xxx.xxx
198.xx.xxx.xxx
198.xx.xxx.xxx Resource Declaration:
198.xx.xxx.xxx
198.xx.xxx.xxx ---------------------
198.xx.xxx.xxx # In /var/chef/cache/cookbooks/rails/recipes/production.rb
198.xx.xxx.xxx
198.xx.xxx.xxx  40: template node[:rails][:app_root]+"/shared/config/database.yml" do
198.xx.xxx.xxx  41:     owner "root"
198.xx.xxx.xxx  42:     group "xx-dev"
198.xx.xxx.xxx  43:     mode 0775
198.xx.xxx.xxx  44: end
198.xx.xxx.xxx  45:
198.xx.xxx.xxx
198.xx.xxx.xxx Compiled Resource:
198.xx.xxx.xxx
198.xx.xxx.xxx ------------------
198.xx.xxx.xxx
198.xx.xxx.xxx # Declared in /var/chef/cache/cookbooks/rails/recipes/production.rb:40:in `from_file'
198.xx.xxx.xxx
198.xx.xxx.xxx template("/var/www/mysite.com/shared/config/database.yml") do
198.xx.xxx.xxx   provider Chef::Provider::Template
198.xx.xxx.xxx   action "create"
198.xx.xxx.xxx   retries 0
198.xx.xxx.xxx   retry_delay 2
198.xx.xxx.xxx   path "/var/www/mysite.com/shared/config/database.yml"
198.xx.xxx.xxx   backup 5
198.xx.xxx.xxx   source "database.yml.erb"
198.xx.xxx.xxx   cookbook_name "rails"
198.xx.xxx.xxx   recipe_name "production"
198.xx.xxx.xxx   mode 509
198.xx.xxx.xxx   owner "root"
198.xx.xxx.xxx   group "xx-dev"
198.xx.xxx.xxx end
198.xx.xxx.xxx
198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] ERROR: Running exception handlers
198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json
198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] ERROR: Exception handlers complete
198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] FATAL: Net::HTTPServerException: template[/var/www/mysite.com/shared/config/database.yml] (rails::production line 40) had an error: Net::HTTPServerException: 403 "Forbidden"

How many nodes are you using? – sethvargo May 30 '14 at 21:27 — sethvargo, May 30 '14 at 21:27
I have approximately 50, Mr. Vargo. – Jahkobi Digital Jun 02 '14 at 16:30 — Jahkobi Digital, Jun 02 '14 at 16:30
Is this a long running recipe? – Stuart Feldt Jun 02 '14 at 19:22 — Stuart Feldt, Jun 02 '14 at 19:22
Long running? This is my most extensive cookbook for sure. – Jahkobi Digital Jun 02 '14 at 20:21 — Jahkobi Digital, Jun 02 '14 at 20:21

score 5 · Accepted Answer · answered Jun 12 '14 at 05:03

5

Objects from cookbooks (templates, cookbook_files) are lazily-loaded by default.

If it's a very long run, and it sounds like it is, then the runtime might exceed the maximum time window enforced by the server from the original authentication.

You can turn off lazy loading on the client by adding

no_lazy_load true

to your client.rb. See http://docs.opscode.com/config_rb_client.html

answered Jun 12 '14 at 05:03

Julian Dunn

286
1
4

How can you set this during the bootstrap process? – Chris F Sep 30 '14 at 18:09
This did NOT work for me. I still get the 403 error in spite of no_lazy_load true in my client.rb file – Chris F Sep 30 '14 at 19:27
Are you running your own Chef server? – Julian Dunn Oct 06 '14 at 03:23
3

Julian, yes I am. I found a solution where I had to increase a url timeout value on my server from 900 to 1800 and that worked. – Chris F Oct 08 '14 at 00:40

score 3 · Answer 2 · answered Oct 09 '14 at 19:29

In addition to Julian's answer, this worked for me when we ran recipes that took over 15 minutes. 15 mins is the default TTL value for the Chef server:

Edit app.config file under /var/opt/chef-server/erchef/etc or /opt/opscode/embedded/service/opscode-erchef/etc depending on the version of Chef you have installed.

Set the s3_url_ttl value to something more than 900 seconds. We used 3600 to get around the problem and then rebooted the Chef server host.

Chef-client throws 403 forbidden intermittently

2 Answers2