3

I get a strange error when a chef-client tries to execute remote_resource for a big local file. From stack trace I guess ruby copy files itself. My disk has a lot of free space. Also var and tmp folders has at leas 2 Gbytes. If I do this job myself with cp command or I replace remote_file resource with execute one it's okay.

Chef complains about lack of disk space.

This resource fails for a file of 4G size with message No space on device.

remote_file "/tmp/copy.img" do
    source "file://tmp/origin.img"
    action :create
end

I made workaround with bash resource and it works.

execute "disk-image" do
    command "cp /tmp/origin.img /tmp/copy.img" 
    not_if "cmp /tmp/origin.img /tmp/copy.img"
end
Daniil Iaitskov
  • 5,525
  • 8
  • 39
  • 49
  • 2
    I think we need a better explanation of what you're trying to do. – the Tin Man Nov 14 '13 at 22:35
  • Have you received this error using any other resources or in any other scenario? I have a similar issue on kitchen converge, except immediately following: "Resolving cookbook dependencies with Berkshelf 7.0.8..." – Brandon Hawbaker Dec 27 '19 at 17:43

3 Answers3

1

It's not going to work. remote_file downloads the remote file to somewhere within /var/chef IIRC, then copies to its destination.

Since /var has only 2Gb of space and the file is 4Gb big, it correctly throws the No space left on device error.

cassianoleal
  • 2,556
  • 19
  • 22
  • I cannot agree it's a feature. Intermediate step makes a lot of overhead. It makes remote_file resource ununiversal. I had to override its default provider. But this is a main purpose of resource/provider abstraction. – Daniil Iaitskov Nov 15 '13 at 13:59
  • I didn't say that it's a feature, I said that this is how it works, how it's designed. And it makes sense for **remote files**. I wouldn't like to have an incomplete download lying around where a complete file should be. Either way, my answer is correct and I'd deeply appreciate it if you accepted it as such. :) – cassianoleal Nov 15 '13 at 19:45
  • Oh, and you're of course welcome to file a bug in http://tickets.opscode.com if you think this is one. – cassianoleal Nov 15 '13 at 19:47
  • @DaneelS.Yaitskov You can customize the download location used by chef by setting the file_cache_path setting. See http://docs.opscode.com/config_rb_client.html – Mark O'Connor Nov 15 '13 at 21:58
1

9/20/2016: Chef 12.0 shipped with file_stating_uses_destdir being defaulted to true so this should no longer be an issue (the remote_file but where it streams to /tmp may still exist).

First the real simple statement: If you've got a 4GB file in /tmp and you only have 2GB left in /tmp, then obviously copying the 4GB will fail, and nothing can help you. I'm assuming you have at least 4GB in /tmp and only 2GB left in /var which is the only interesting case to address.

As of 11.6.0 (to 11.10.2 at least) chef-client will create a tempfile using ruby's Tempfile.new and will copy the contents to that temp file and then will mv it into place. The tempfile location will be determined by ENV['TMPDIR'] and that differs based on your O/S distro (e.g. on a Mac that will be something like /var/folders/rs/wqwmj_1n59135dhhbvfdrlqh0001yj/T/ and not just /tmp or even /var/tmp), so it may not be obvious where the intermediate tempfile is created. You may be running into that problem. You should be able to see from the chef-client -l debug output what tempfile location chef is using and if you df -k that directory you might see that it is 100%.

Also, look at df -i to see if you've run out of inodes somehow which will also throw a no space left on device error.

You can set chef-client globally to use the destination directory as the tmpdir for creating files via adding this to client.rb:

file_staging_uses_destdir true

Then if your destination dir is '/tmp' the tempfile will get created there and then will simply get renamed in the directory in order to deploy it. That ensures that if there's enough space on the target device to hold the result, then the resource should always succeed to write the tempfile. It also avoids the problem if /tmp and the destdir are on different filesystems that the mv to rename and deploy the file will get translated into a copy-and-unlink-src operation which can fail in several different ways.

The answer by @cassianoleal is not correct in stating that chef-client always uses /var/cache as a temp location. Changing file_cache_path will also not have an effect. That is confusing a common pattern of downloading remote_files into the Chef file_cache_path directory for how remote_file works internally -- those are not the same thing. There is no file_cache_path in the question, so there should not be any file_cache_path in the answer.

The behavior of remote_file with file:// URLs is a bit round-a-bout, but that is because they're necessary for all other URLs (as @cassianoleal correctly mentioned). The behavior with file_staging_uses_destdir is probably correct, however, since you do want to take into account edge conditions where you run out of room and truncate the file or the server crashes in the middle of a copy operation and you don't want a half-populated file left over. By writing to a tempfile and closing it and then renaming a lot of those edge conditions are avoided.

lamont
  • 3,854
  • 1
  • 20
  • 26
1

Thank you @lamont for the explanation. To cut to the chase a bit, the only solution that worked for me was to add the following to my Chef recipe, prior to any calls to remote_file:

ENV['TMP'] = '/mytmp'
ENV['TMPDIR'] = '/mytmp'

where /mytmp is a directory on a volume with enough space to hold my file.

The promising feature of adding:

file_staging_uses_destdir true

to /etc/chef/client.rb currently does not work, due to this bug: https://tickets.opscode.com/browse/CHEF-5311.

dr_zeus
  • 31
  • 4