Why would Puppet exclude a file resource without error?

Question

The oddest thing is happening during this Puppet run (using Puppet Apply) and it has me perplexed. I've tried my usual channels to find solutions but I cannot find anything on the topic. I apologize for being so verbose, I just don't want to leave anything out :\

-

My Class

class c2c::profile::app::logio::stage_support {

  # Log.io plugin
  # Create plugin directories
  exec { "create_codec_dir":
    command => "/bin/mkdir --parents /etc/logstash/plugins/logstash/codecs --mode=0775",
    creates => '/etc/logstash/plugins/logstash/codecs',
  }

  # Install plugin
  file { "logio_plugin_file":

    # update: added in response to a serverfault comment
    ensure  => file,

    path    => '/etc/logstash/plugins/logstash/codecs/logio.rb',
    mode    => '0775',
    owner   => 'root',
    group   => 'root',
    source  => 'puppet:///modules/c2c/logstash/logio_codec.rb',
    require => Exec['create_codec_dir'],
    notify  => Service['logstash'],
  }

}

.. and my Puppet run is skipping the logio_plugin_file instruction (or, for whatever reason, is not copying the file).

-

Logs, Etc

The debug log is rather large, and did not want to paste the whole thing, so here is every line in the Puppet run that mentions 'logio_plugin_file', but I'll gladly provide more if needed.

Debug: /File[logio_plugin_file]/seluser: Found seluser default 'system_u' for /etc/logstash/plugins/logstash/codecs/logio.rb
Debug: /File[logio_plugin_file]/selrole: Found selrole default 'object_r' for /etc/logstash/plugins/logstash/codecs/logio.rb
Debug: /File[logio_plugin_file]/seltype: Found seltype default 'etc_t' for /etc/logstash/plugins/logstash/codecs/logio.rb
Debug: /File[logio_plugin_file]/selrange: Found selrange default 's0' for /etc/logstash/plugins/logstash/codecs/logio.rb
..
Debug: /File[logio_plugin_file]/require: requires Exec[create_codec_dir]
Debug: /File[logio_plugin_file]/notify: subscribes to Service[logstash]
..
Debug: /File[logio_plugin_file]: Autorequiring File[logstash_codec_plugins]

/var/lib/puppet/state/last_run_report.yaml

The term 'logio_plugin_file' is only mentioned as part of two Puppet::Util::Log entries, and no File[logio_plugin_file] entry exists.

Below is a quick snippet of those two entries:

- !ruby/object:Puppet::Util::Log
  level: !ruby/sym debug
  time: 2013-12-06 17:30:38.009095 +00:00
  tags: 
    - debug
    - file
    - logio_plugin_file
    - class
    ... (25 more ) ...
  line: 41
  source: /File[logio_plugin_file]/require
  file: /tmp/vagrant-puppet/modules-0/c2c/manifests/profile/app/logio/stage_support.pp
  message: "requires Exec[create_codec_dir]"

  .. further down, mostly the same except ..

  message: "subscribes to Service[logstash]"

-

Strangeness

This file is included within a large catalog, with dozens or hundreds of other classes, but what's weird is if I include this class directly using a quick testing manifest (also using puppet apply), it works like a charm. (Although I did have to remove the service requirement)

node default {
  class { 'c2c::profile::app::logio::stage_support': }
}

.. and I get this in my output ..

Notice: /File[logio_plugin_file]/ensure: defined content as '{md5}41d00952843b8159b95ce4fcd8015cda'

.. and this in last_run_report.yaml ..

File[logio_plugin_file]: !ruby/object:Puppet::Resource::Status
  resource: File[logio_plugin_file]
  file: /tmp/vagrant-puppet/modules-0/c2c/manifests/profile/app/logio/stage_support.pp
  line: 40
  evaluation_time: 0.023071
  change_count: 1
  out_of_sync_count: 1
  ...

-

Additional Points, Info, and Tests

• I only just added the Notify => Service['logstash'], and this behavior was occuring before I did so.
• No errors are emitted
• I added a few notify{} calls in my stage_support class to ensure it was being properly included in the catalog, and it is.
• Exec['create_codec_dir'] is creating my directory
• I tried changing the path param to logio2.rb to see if the class
• I tried requiring File['logio_plugin_file'] from another resource. The requiring resource ran, but the file was not created.
• Update: I tried reducing my file type call to only include the title plus ensure, path, and source, which had no effect.

• Update: I tried renaming the resource, (e.g. logio_plugin_file_x), which did not help.

  [root@dev ~]# puppet --version
  3.2.3
  
  [root@dev ~]# facter --version
  1.7.2
  
  [root@dev ~]# cat /etc/redhat-release
  CentOS release 6.4 (Final)
  

-

Important Note

This problem comes on the tail end of me doing a code restructuring, so that's almost certainly to blame, but I am not able to troubleshoot this.

Any help is much appreciated!

Answer 1

As expected, the source of the problem turned out to be simple, though I still do not understand why Puppet did not fire an error about this:

profile/logstash.pp: (elsewhere)

file { 'logstash_plugin_sub':
  path      => '/etc/logstash/plugins/logstash',
  ensure    => 'directory',
  owner     => 'root',
  group     => 'root',
  mode      => '0775',
  require   => [ File['logstash_plugin_sub'] ]
}

Note the circular reference in the require: File['logstash_plugin_sub'] -> File['logstash_plugin_sub']

The resulting behavior is very weird, and I found the problem using this test:

file { "logio_plugin_file_a":
  path    => '/etc/logstash/logio.rb',
  source  => 'puppet:///modules/c2c/logstash/logio_codec.rb',
}
file { "logio_plugin_file_b":
  path    => '/etc/logstash/plugins/logio.rb',
  source  => 'puppet:///modules/c2c/logstash/logio_codec.rb',
}
file { "logio_plugin_file_c":
  path    => '/etc/logstash/plugins/logstash/logio.rb',
  source  => 'puppet:///modules/c2c/logstash/logio_codec.rb',
}
file { "logio_plugin_file_d":
  path    => '/etc/logstash/plugins/logstash/codecs/logio.rb',
  source  => 'puppet:///modules/c2c/logstash/logio_codec.rb',
}

The idea being, principally, try adding my file at deeper and deeper locations in the tree. This idea came to me as a result of being able to change the path in my original example, and achieve success.

So, the following was created:

/etc/logstash/logio.rb
/etc/logstash/plugins/logio.rb

But the two deeper resources were skipped, so I knew my problem centered around /etc/logstash/plugins/logstash. Upon close inspection, I found the circular reference.

Hope this helps someone, and thank you to Shane Madden for his time.