Vagrant's puppet provisioner not reading hiera.yaml backend correctly?

Question

I have a puppet problem that is specific to the way that Vagrant uses the manifests/modules/hiera-config with its puppet provisioner, since a "puppet apply site.pp" works fine in the resultant vagrant deployed VM (when applied locally in the guest OS itself). In the directory with the Vagrantfile I have a "puppet_files" subdirectory with the manifests, modules, and hiera files that will be copied to /etc/puppet on the VM (I use a puppet module with "file" directives to copy those files there).

My host environment is OSX and I am using vagrant to deploy Centos 6 on VirtualBox.

Background Information:

When I type "vagrant up" I see this to begin with:

==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Setting hostname...
==> default: Mounting shared folders...
    default: /vagrant => /Users/juser/vm_stuff/vagrant-fresh
    default: /tmp/vagrant-puppet-1/manifests => /Users/juser/vm_stuff/vagrant-fresh/puppet_files/manifests
    default: /tmp/vagrant-puppet-1/modules-0 => /Users/juser/vm_stuff/vagrant-fresh/puppet_files/modules
==> default: Running provisioner: shell...

So it looks like it creates temporary files on my OSX filesystem and copies the source directories/files from where I specified in the Vagrantfile. On the VM itself the puppet directory is appropriately mounted as /vagrant/puppet_files. Here is the relevant section of the Vagrantfile for the puppet config:

config.vm.provision "puppet" do |puppet|
    puppet.manifests_path = "puppet_files/manifests"
    puppet.module_path    = "puppet_files/modules"
    puppet.hiera_config_path = "puppet_files/hiera_config/hiera.yaml"
    puppet.manifest_file  = "site.pp"
    puppet.options = "--verbose --debug"
  end

The site.pp has only two lines of importance (calling two modules):

include ::hierasetup
include ::jboss

And the hiera.yaml file looks like this:

:backends:
        - json

:logger: console

:hierarchy:
        - "node/%{::fqdn}"
        - common

:json:
    :datadir: '/etc/puppet/hieradata/'

And my hierasetup module (called in site.pp) also copies a json hiera file to /etc/puppet/hieradata/common.json. And just FYI the jboss module is the module that tries to use hiera with a "hiera_hash" and "create_resources" (which works fine when applied locally/manually in the Linux VM).

The Problem:

Vagrant has no problem importing the manifests and modules and even applying the manifest, but it constantly fails to read my hiera.yaml file (correctly?), because if it did it would see that I specified json and not yaml as the backend:

Debug: importing '/tmp/vagrant-puppet-1/modules-0/hierasetup/manifests/init.pp' in environment production
Debug: Automatically imported hierasetup from hierasetup into production
Debug: importing '/tmp/vagrant-puppet-1/modules-0/jboss/manifests/init.pp' in environment production
Debug: Automatically imported jboss from jboss into production
Debug: hiera(): Hiera YAML backend starting
Debug: hiera(): Looking up jbossas in YAML backend
Debug: hiera(): Looking for data source common
Debug: hiera(): Cannot find datafile /var/lib/hiera/common.yaml, skipping

Error: create_resources(): second argument must be a hash at /tmp/vagrant-puppet-1/modules-0/jboss/manifests/init.pp:14 on node josh-new.morgan.haib.org
Wrapped exception:
create_resources(): second argument must be a hash

Error: create_resources(): second argument must be a hash at /tmp/vagrant-puppet-1/modules-0/jboss/manifests/init.pp:14 on node josh-new.morgan.haib.org
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

puppet apply --verbose --debug --modulepath '/tmp/vagrant-puppet-1/modules-0:/etc/puppet/modules' --hiera_config=/tmp/vagrant-puppet-1/hiera.yaml --manifestdir /tmp/vagrant-puppet-1/manifests --detailed-exitcodes /tmp/vagrant-puppet-1/manifests/site.pp || [ $? -eq 2 ]

Why is it looking for common.yaml and not /etc/puppet/hieradata/common.json? Why does it think it is a YAML backend and not JSON? It is not reading my hiera.yaml file (which I gave the correct relative path to in the Vagrantfile) and operating on some kind of defaults?

Bottom line: The "puppet apply site.pp" works perfectly if I'm in the Linux environment on the VM applying it locally, but it does not work "the Vagrant way" with the puppet provisioner. I must be missing something about how the hiera config works for Vagrant.

@ShaneMadden So the actual file lives in /Users//vm_stuff/vagrant-fresh/puppet_files/hiera_config/ and the point of the Vagrant "puppet.hiera_config_path" parameter (as I take it) is to point it to your own custom location. Part of Vagrant's own internal deployment methodology (again, as I take it) is to COPY my files to some temporary location of its choosing which I have no control over (and which I have not found documented as some standard location under /tmp). — SeligkeitIstInGott, Jun 19 '14 at 13:41
continued: For example init.pp does not actually live at /tmp/vagrant-puppet-1/modules-0/hierasetup/manifests/init.pp but rather /Users//vm_stuff/vagrant-fresh/puppet_files/manifests/init.pp but it copies it there at run time because of "puppet.module_path". I expect it to be able to handle hiera.yaml in the same manner. Also note (and this confused me for a while) it is pointing to /tmp on my local OSX machine and not /tmp on the VM itself (though I think maybe the VM also gets a copy of the subdirs created under /tmp???). No clue. This is why I'm confused as to how exactly it works. — SeligkeitIstInGott, Jun 19 '14 at 13:43
My edit window expired for the above comment. Correction: "init.pp does not actually live at /tmp/vagrant-puppet-1/modules-0/hierasetup/manifests/init.pp but rather /Users//vm_stuff/vagrant-fresh/puppet_files/modules/hierasetup/manifests/init.pp". I copy and pasted the wrong path under /Users above. — SeligkeitIstInGott, Jun 19 '14 at 13:53
Hmmm, but this line in my "vagrant up" output would seem to indicate the /tmp directory is configurable in some way (since it says "default"): "default: /tmp/vagrant-puppet-1/manifests => /Users/juser/vm_stuff/vagrant-fresh/puppet_files/manifests" But I didn't see one of those messages for the hiera directory/file. — SeligkeitIstInGott, Jun 19 '14 at 14:04
The example in the [documentation](https://docs.vagrantup.com/v2/provisioning/puppet_apply.html) uses `puppet.hiera_config_path` of just `hiera.yaml` and a `puppet.working_directory` with a fully qualified path to the puppet directory (in their case `/tmp/vagrant-puppet`) - maybe try mimicking that and see if it's any more successful? Seems like the handling of the hiera config file is inconsistent with the handling of the other files. — Shane Madden, Jun 19 '14 at 16:10
I'll try that. In the meantime I found a similar SE question: http://stackoverflow.com/questions/17956781/puppet-looking-for-hiera-yaml-in-the-wrong-place. The answer by ivanlei there is still confusing though since he sets the working_directory to his local OSX path, while the book he links to sets it to the (mounted) /vagrant path in the VM. But I may indeed have to explicitly set the working_directory. — SeligkeitIstInGott, Jun 19 '14 at 19:56
It appears that the hiera.yaml file will always be copied to the /tmp/vagrant-puppet(-X) directory when puppet.hiera_config_path is used, unless manually overwritten with puppet.options "--hiera_config=/some/other/path". — SeligkeitIstInGott, Jun 19 '14 at 22:43
To answer "Why does it think it is a YAML backend and not JSON?", it was simply that when the jboss module was calling hiera, because hiera.yaml wasn't present to configure it otherwise, it tried to look (following default behavior) for a YAML hiera backend and failed since the files were absent. — SeligkeitIstInGott, Jun 20 '14 at 14:42

SeligkeitIstInGott · Accepted Answer · 2014-06-19T23:41:13.750

Okay so it appears I ran into two different problems. The largest problem was that although I had a module that I created called "hieraconfig" which would copy my precreated hiera.yaml and common.json files to /etc/puppet, the jboss module which calls hiera was being executed first (although I included it after hieraconfig in manifest - see below). I tried to fix this to evaluate hierasetup first in the site.pp manifest, but it still ran the jboss module first:

stage { 'pre':
  before => Stage['main']
}

# add the hierasetup module to the new 'pre' run stage
class { 'hierasetup':
  stage => 'pre'
}

include ::hierasetup
include ::jboss

I have no current fix for this. But at least I know that part of the failure was that /etc/puppet/hiera.yaml and /etc/puppet/hieradata/common.json were not even present on the VM when hiera was being called from the jboss module, since hierasetup had not run yet. I made this work temporarily by using Vagrant's script provisioner instead of puppet to copy the files there for me.

The second issue was that because I thought those files were present the whole time I got quite confused about what "puppet.hiera_config_path" was supposed to do. Through various trial and error attempts here is what I have discovered:

puppet.manifests_path, puppet.modules_path, and puppet.hiera_config_path only point to paths on the host machine (in my case on OSX). Don't be fooled into thinking, if you use relative paths, that it is relative to the /vagrant mount on the VM, and not rather relative to the directory containing your Vagrantfile on the host machine (though that is obviously what gets mounted to /vagrant).
Each of those _path variables, IF and ONLY IF they are actually set in the Vagrantfile (otherwise default puppet directories will be searched), will cause Vagrant to copy the modules, manifests, and hiera config to subdirectories under /tmp/vagrant-puppet[-X] (with -X possibly being an additional number suffix like /tmp/vagrant-puppet-1) and tell puppet to look under /tmp for them.

If you have your puppet files already on the VM that you want to use but set the _path variables then Vagrant's puppet command is not going to find them, because it redirects puppet to /tmp/vagrant-puppet[-X] when you set them. You cannot change this destination on the VM, although the "vagrant up" output will notify you where they are being mapped like this:

default: /tmp/vagrant-puppet-1/manifests => /Users//vm_stuff/vagrant-fresh/puppet_files/manifests

default: /tmp/vagrant-puppet-1/modules-0 => /Users//vm_stuff/vagrant-fresh/puppet_files/modules

The path after the => arrow is the source directory on your host machine which you specified with the _path variables in the Vagrantfile. The path before the => arrow is the location on the VM where the source directory contents will be copied.

If you instead prefer Vagrant to have puppet look in default directories for files you have already placed on the VM (which you do not need copied by Vagrant for you into /tmp) then do not specify any _path variables in Vagrant. Alternately if neither the default locations or the /tmp location are satisfactory you may completely override the default Vagrant behavior by explicitly specifying where to look for each with the puppet.options variable, like this:

puppet.options = "--hiera_config=/path/to/hiera.yaml --modulepath '/path/to/modules/ --manifestdir /path/to/manifests/"

If anything fails and you have set --verbose && --debug as part of the puppet.options (possibly it will show without those set as well) you will get an error showing the actual puppet command sent via ssh to the guest VM, like this:

The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

puppet apply --verbose --debug --modulepath '/tmp/vagrant-puppet-1/modules-0:/etc/puppet/modules' --manifestdir /tmp/vagrant-puppet-1/manifests --hiera_config=/tmp/vagrant-puppet-1/hiera.yaml --detailed-exitcodes /tmp/vagrant-puppet-1/manifests/site.pp || [ $? -eq 2 ]

As long as --hiera_config is pointing to the correct location/file in this command it should read your configuration with no problems. I had an issue temporarily (before I discovered that /etc/puppet/hieradata/common.json was not present on the VM) to where it would read the hiera.yaml but then fail due to common.json being absent.

All said and done though hiera looks in very specific locations depending on whether you set "puppet.hiera_config_path" or not, unless you manually override the puppet.options.

Vagrant's puppet provisioner not reading hiera.yaml backend correctly?

1 Answers1