Chef Kitchen converge is generating [No space left on device - write] error

Question

On a Windows 10 host, vagrant, and a local virtual box VM (Windows Server 2012 R2), I suddenly began seeing the following error when performing a kitchen converge:

-----> Starting Kitchen (v2.2.5)
$$$$$$ Deprecated configuration detected:
chef_omnibus_url
Run 'kitchen doctor' for details.

-----> Converging <default-BOX>...
       Preparing files for transfer
       Preparing dna.json
       Resolving cookbook dependencies with Berkshelf 7.0.8...
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Failed to complete #converge action: [No space left on device - write] on default-BOX
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

This isn't the full kitchen.yml, but here's the provision settings showing the version of the chef client:

provisioner:
  name: chef_zero
  # You may wish to disable always updating cookbooks in CI or other testing environments.
  # For example:
  #   always_update_cookbooks: <%= !ENV['CI'] %>
  always_update_cookbooks: true
  require_chef_omnibus: true
  product_name: chef
  product_version: 14.12.9
  install_strategy: once
  platform: windows
  platform_version: 10.0
  architecture: x86_64

The log suggests there's some kind of copy error with berkshelf:

[2019-12-30T16:17:11.213316 #6088] ERROR -- Kitchen: Message: Failed to complete #converge action: [No space left on device - write] on default-BPA-DEV
E, [2019-12-30T16:17:11.213350 #6088] ERROR -- Kitchen: ----------------------
E, [2019-12-30T16:17:11.213384 #6088] ERROR -- Kitchen: ------Backtrace-------
E, [2019-12-30T16:17:11.213419 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/fileutils.rb:1387:in `copy_stream'
E, [2019-12-30T16:17:11.213455 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/fileutils.rb:1387:in `block (2 levels) in copy_file'
E, [2019-12-30T16:17:11.213489 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/fileutils.rb:1386:in `open'
E, [2019-12-30T16:17:11.213568 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/fileutils.rb:1386:in `block in copy_file'
E, [2019-12-30T16:17:11.213608 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/fileutils.rb:1385:in `open'
E, [2019-12-30T16:17:11.213644 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/fileutils.rb:1385:in `copy_file'
E, [2019-12-30T16:17:11.213679 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/fileutils.rb:492:in `copy_file'
E, [2019-12-30T16:17:11.213715 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/fileutils.rb:419:in `block in cp'
E, [2019-12-30T16:17:11.213751 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/fileutils.rb:1557:in `block in fu_each_src_dest'
E, [2019-12-30T16:17:11.213786 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/fileutils.rb:1573:in `fu_each_src_dest0'
E, [2019-12-30T16:17:11.213820 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/fileutils.rb:1555:in `fu_each_src_dest'
E, [2019-12-30T16:17:11.213855 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/fileutils.rb:418:in `cp'
E, [2019-12-30T16:17:11.213890 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/berkshelf-7.0.8/lib/berkshelf/file_syncer.rb:95:in `block in sync'
E, [2019-12-30T16:17:11.213924 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/berkshelf-7.0.8/lib/berkshelf/file_syncer.rb:78:in `each'
E, [2019-12-30T16:17:11.213959 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/berkshelf-7.0.8/lib/berkshelf/file_syncer.rb:78:in `sync'
E, [2019-12-30T16:17:11.213994 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/berkshelf-7.0.8/lib/berkshelf/berksfile.rb:688:in `block in vendor'
E, [2019-12-30T16:17:11.214029 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/tmpdir.rb:93:in `mktmpdir'
E, [2019-12-30T16:17:11.214064 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/berkshelf-7.0.8/lib/berkshelf/berksfile.rb:635:in `vendor'
E, [2019-12-30T16:17:11.214214 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/provisioner/chef/berkshelf.rb:66:in `block in resolve'
E, [2019-12-30T16:17:11.214258 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/thor-0.20.3/lib/thor/shell/basic.rb:22:in `mute'
E, [2019-12-30T16:17:11.214294 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/provisioner/chef/berkshelf.rb:61:in `resolve'
E, [2019-12-30T16:17:11.214329 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/provisioner/chef/common_sandbox.rb:318:in `block in resolve_with_berkshelf'
E, [2019-12-30T16:17:11.214364 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/provisioner/chef/common_sandbox.rb:315:in `synchronize'
E, [2019-12-30T16:17:11.214408 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/provisioner/chef/common_sandbox.rb:315:in `resolve_with_berkshelf'
E, [2019-12-30T16:17:11.214522 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/provisioner/chef/common_sandbox.rb:252:in `prepare_cookbooks'
E, [2019-12-30T16:17:11.214657 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/provisioner/chef/common_sandbox.rb:48:in `populate'
E, [2019-12-30T16:17:11.214739 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/provisioner/chef_base.rb:289:in `create_sandbox'
E, [2019-12-30T16:17:11.214789 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/provisioner/chef_zero.rb:50:in `create_sandbox'
E, [2019-12-30T16:17:11.214831 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/provisioner/base.rb:71:in `call'
E, [2019-12-30T16:17:11.214869 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/instance.rb:409:in `block in converge_action'
E, [2019-12-30T16:17:11.214906 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/instance.rb:552:in `synchronize_or_call'
E, [2019-12-30T16:17:11.214942 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/instance.rb:514:in `block in action'
E, [2019-12-30T16:17:11.215029 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/2.6.0/benchmark.rb:293:in `measure'
E, [2019-12-30T16:17:11.215069 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/instance.rb:513:in `action'
E, [2019-12-30T16:17:11.215105 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/instance.rb:404:in `converge_action'
E, [2019-12-30T16:17:11.215141 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/instance.rb:382:in `block (2 levels) in transition_to'
E, [2019-12-30T16:17:11.215176 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/lifecycle_hooks.rb:45:in `run_with_hooks'
E, [2019-12-30T16:17:11.215211 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/instance.rb:381:in `block in transition_to'
E, [2019-12-30T16:17:11.215248 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/instance.rb:380:in `each'
E, [2019-12-30T16:17:11.215284 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/instance.rb:380:in `transition_to'
E, [2019-12-30T16:17:11.215318 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/instance.rb:140:in `converge'
E, [2019-12-30T16:17:11.215354 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/command.rb:197:in `public_send'
E, [2019-12-30T16:17:11.215390 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/command.rb:197:in `run_action_in_thread'
E, [2019-12-30T16:17:11.215424 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/test-kitchen-2.2.5/lib/kitchen/command.rb:169:in `block (2 levels) in run_action'
E, [2019-12-30T16:17:11.215459 #6088] ERROR -- Kitchen: C:/opscode/chefdk/embedded/lib/ruby/gems/2.6.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
E, [2019-12-30T16:17:11.215494 #6088] ERROR -- Kitchen: ----End Backtrace-----

In the past I had an issue where it would hang on the berkshelf (at least appeared to) and then I would reboot the VM and the converge would work. It almost looks like a similar issue except the reboot isn't resolving the issue.

No. I have tried recreating the entire target machine and then it works the first time, but afterwords fails for subsequent converges. — Brandon Hawbaker, Jan 14 '20 at 18:28

score 1 · Answer 1 · answered Jan 03 '20 at 05:29

1

i think that the issue that you are experiencing has nothing to do with kitchen directly, but rather with the image of your virtual-machine, which does not have enough disk space.

No space left on device - write] on default-BPA-DEV

this happens when someone created a very small storage device when upon the virtual-machine creation, and then derive a virtual-machine image out of it.

but you are lucky, since you can use vagrant-disksize to dynamically increase the virtual-machine disk size in conjenction with vagrantfiles driver option for kitchen-vagrant.

this is how i fixed the same issue :)

answered Jan 03 '20 at 05:29

Mr.

9,429
13
58
82

This was not the issue as all my disk drives were substantially larger than required both on the Virtual Box side and in Windows. What I did to resolve the issue proves this fact. – Brandon Hawbaker Jan 14 '20 at 18:29
One important fact about when this issue occurs is only when using Berkshelf. – Brandon Hawbaker Jan 14 '20 at 18:38

score 0 · Answer 2 · answered Jan 14 '20 at 18:36

0

What I eventually did to resolve this issue was to stop using Berkshelf altogether. I switched over to using policy files instead of Berkshelf for managing package dependencies within the cookbook and the issue went away.

https://docs.chef.io/policyfile.html#chef-generate-policyfile

chef generate policyfile

This creates a file like "Policyfile.rb" in your cookbook. Once you have the policy file, kitchen may complain about the Policyfile.lock.json. You may have to procure this file as well. I believe I copied mine from another cookbook, after which, the JSON was automatically updated with the proper structure using kitchen.

answered Jan 14 '20 at 18:36

Brandon Hawbaker

158
1
7

strange. can you share a repository to reproduce the issue? – Mr. Jan 16 '20 at 12:36
I can't share my code base in this case, because it's not mine. If you have specific questions though I can remove proprietary information. – Brandon Hawbaker Jan 28 '20 at 18:39

Chef Kitchen converge is generating [No space left on device - write] error

2 Answers2