0

Learning Puppet and Hiera and I've run into a roadblock. I apologize in advance if this is something simple. Given the following files within my GitLab for the PuppetClass es_strat:

hiera.yaml

    ---
version: 5
defaults:
  data_hash: yaml_data
  datadir: data
hierarchy:
  - name: Hostname
    path: "hosts/%{facts.fqdn}.yaml"
  - name: hostgroup and environments
    path: "hostgroups/%{::hostgroup}/environments/%{facts.env}%{facts.env_num}.yaml"
  - name: hostgroup and tier
    path: "hostgroups/%{::hostgroup}/tiers/%{facts.tier}.yaml"
  - name: hostgroup
    path: "hostgroups/%{::hostgroup}.yaml"
  - name: tier
    path: "tiers/%{facts.tier}.yaml"
  - name: Common
    path: common.yaml

common.yaml

    ---
es_strat::es_heap     : 16g
es_strat::es_version  : 2.3.2
es_strat::kopf_version: v2.1.2
es_strat::java_version: jdk1.7.0_91
es_strat::es_instances: 
  "%{::hostname}": 
    config: 
      bootstrap.mlockall: "true" 
      cluster.name: "%{::datacenter}%{::env}%{::env_num}stratsrch"
      discovery.zen.ping: 
        multicast: 
          enabled: "false"
        unicast: 
          hosts: "%{es_masters}"
      http: 
        compression: "true"
        enabled: "true"
        max_content_length: 500mb
        port: "9200"
      network.publish_host: "%{::ipaddress}"
      network.host: "%{::ipaddress}"
      node: 
        data: "true"
        master: "true"
        name: "%{::hostname}"
      path.logs: /indexes/logs
      transport.tcp.compress: "true"
      transport.tcp.port: "9300"
      indices.store.throttle.type: none
      script:
        indexed: "true"
        udpate: "true"
    datadir: /indexes/data

init.pp

# Class: es_strat
#
# This module manages es_strat
#
# Parameters: none
#
# Actions:
#
# Requires: see Modulefile
#
# Sample Usage:
#
class es_strat (
  $es_heap      = hiera('es_strat::es_heap'),
  $es_instances = hiera('es_strat::es_instances'),
  $es_version   = hiera('es_strat::es_version'),
  $java_version = hiera('es_strat::java_version'),
  $es_hosts     = hiera('es_strat::es_hosts', undef),
  $kopf_version = hiera('es_strat::kopf_version', undef),
  $es_scripts   = hiera('es_strat::es_scripts', undef),
){
  # Create Elasticsearch user with reserved UID/GID.
  # TODO: Move this to virtual::users module
  ensure_resource('group', 'elasticsearch', {
    ensure     => 'present',
    forcelocal => true,
    gid        => 668981,
    before     => User['elasticsearch']
  })
  ensure_resource('user', 'elasticsearch', {
    ensure     => 'present',
    comment    => 'elasticsearch user',
    forcelocal => true,
    home       => '/opt/elasticsearch',
    shell      => '/bin/false',
    uid        => 3160070,
    gid        => 668981,
  })
  # Ensure elasticsearch logs are writeable. 
    file { [
    '/indexes/',
    '/indexes/logs',
  ]:
    ensure => directory,
    owner  => 'elasticsearch',
  }
  # Define master hosts to connect to. 
  if ! $es_hosts {
    $query_es_nodes = query_nodes("(class['es_strat'] and env=${::env} and env_num='${::env_num}')")
    $es_masters = parsejson(inline_template("[<%= @query_es_nodes.map{
      |host|
        \"\\\"\" + host + \":9300\\\"\"
      }.flatten.join(', ')
      %>]"
    ))
  }
  else {
    $es_masters = $es_hosts
  }
  # Install elasticsearch and setup instances. 
  class  { '::elasticsearch':
    version       => $es_version,
    init_defaults => {
      'ES_HEAP_SIZE' => $es_heap,
      'JAVA_HOME'    => "/opt/java/${java_version}/"
    },
    # Look these up again so es_masters will be included.
    instances     => hiera('es_strat::es_instances'),
  }
  # Install plugin if defined. 
  if $kopf_version {
    elasticsearch::plugin { "lmenezes/elasticsearch-kopf/${kopf_version}":
      instances  => $::hostname,
      proxy_host => 'repos.gspt.net',
      proxy_port => 3128
    }
  }
  # Install scripts if defined. 
  if $es_scripts {
    create_resources(elasticsearch::script, $es_scripts)
  }
  # Setup Java in path so plugins work propperly. 
  # TODO Remove this once this bug is fixed. https://github.com/elastic/puppet-elasticsearch/issues/619
  file {'/etc/sysconfig/mcollective':
    content => "export JAVA_HOME=/opt/java/${java_version}/",
    notify  => Service['mcollective'],
  }
}

And then, within Foreman, I have set the following for the Host:

es_heap=hiera("es_strat::es_heap")

es_instances=hiera("es_strat::es_instances")

es_version=hiera("es_strat::es_version")

java_version=hiera("es_strat::java_version")

However, when I run puppet on the Host (specifically: puppet agent -t --no-noop) I receive the following error:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Function lookup() did not find a value for the name 'es_strat::es_instances' on node

Pulling my hair out because it seems like it should be able to get the value from Hiera. Any/all help is greatly appreciated.

Tronyx
  • 1
  • 2
  • Did you restart Foreman after deploying `hiera.yaml` to it? Not sure about v5 but in older versions changes to that file don't get picked up until restart. – shearn89 Oct 04 '17 at 09:18
  • Other Puppet classes using Hiera calls are not having any issues at this time. I will check the age of the Foreman process to see if it's older or newer than the last commit of the hiera.yaml for this class. – Tronyx Oct 04 '17 at 11:45
  • If nothing else has problems then it's probably something syntax-y related to the 'es_instances' entry. Puppet will auto-lookup parameters so you might be able to avoid some config? https://docs.puppet.com/puppet/4.10/hiera_automatic.html – shearn89 Oct 04 '17 at 13:15
  • When I remove the values from within Foreman I get an error that it can't find the value of 'es_strat::es_instances' within the manifests/init.pp file. It's almost like the common.yaml file is not being read, but it only complains about es_instance and not, say, es_heap, which is defined before es_instance is – Tronyx Oct 04 '17 at 17:10
  • Running a --debug --compile on the Puppet Master doesn't show anything about common.yaml, es_instances, or even es_strat. Seems like the class isn't being loaded? – Tronyx Oct 04 '17 at 18:14
  • Is hiera.yaml in /etc/puppet/ on Foreman? – shearn89 Oct 05 '17 at 14:05
  • Yes, @shearn89, it is. – Tronyx Oct 05 '17 at 14:23
  • I'm out of ideas! :( – shearn89 Oct 05 '17 at 15:12
  • @shearn89 - See the answer I posted. Thanks for your help! – Tronyx Oct 06 '17 at 16:06

1 Answers1

0

Ok, so this class had become broken with the update from Puppet 3 to Puppet 4. We use GitLab to control our modules/classes. With that, I was able to get it 99% functional with the following setup:

data/common.yaml

---
es_strat::es_heap     : 16g
es_strat::es_version  : 2.3.2
es_strat::kopf_version: v2.1.2
es_strat::java_version: jdk1.7.0_91
es_strat::es_instances:
  "%{::hostname}":
    config:
      bootstrap:
        mlockall: true
      cluster:
        name: "%{::datacenter}%{::env}%{::env_num}stratsrch"
      discovery:
        zen:
          ping:
            multicast:
              enabled: false
            unicast:
              hosts: "%{es_masters}"
              #hosts: "[]"
      http:
        compression: true
        enabled: true
        max_content_length: 500mb
        port: 9200
      indices:
        store:
          throttle:
            type: none
      network:
        host: "%{::ipaddress}"
        publish_host: "%{::ipaddress}"
      node:
        data: true
        master: true
        name: "%{::hostname}"
      path:
        data: /indexes/data
        logs: /indexes/logs
        repo: /nfs/lvs/elasticsearch/snapshots/stratsrch
      script:
        indexed: true
        udpate: true
      transport:
        tcp:
          compress: true
          port: 9300


#    datadir: /indexes/data
#"es_strat::es_scripts":
#  test:
#    source: "puppet:///modules/es_strat/%{::tier}/test.groovy"

manifests/init.pp

# Class: es_strat
#
# This module manages es_strat
#
# Parameters: none
#
# Actions:
#
# Requires: see Modulefile
#
# Sample Usage:
#
class es_strat (
  $es_heap      = $::es_strat::es_heap,
  $es_instances = undef,
  $es_version   = $::es_strat::es_version,
  $java_version = $::es_strat::java_version,
  $es_hosts     = undef,
  $kopf_version = $::es_strat::kopf_version,
  $es_scripts   = undef,
){

  # Create Elasticsearch user with reserved UID/GID.
  # TODO: Move this to virtual::users module
  ensure_resource('group', 'elasticsearch', {
    ensure     => 'present',
    forcelocal => true,
    gid        => 668981,
    before     => User['elasticsearch']
  })
  ensure_resource('user', 'elasticsearch', {
    ensure     => 'present',
    comment    => 'elasticsearch user',
    forcelocal => true,
    home       => '/opt/elasticsearch',
    shell      => '/bin/false',
    uid        => 3160070,
    gid        => 668981,
  })

  # Ensure elasticsearch logs are writeable.
    file { [
    '/indexes/',
    '/indexes/logs',
  ]:
    ensure => directory,
    owner  => 'elasticsearch',
  }

  # Define master hosts to connect to.
  if ! $es_hosts {
    $query_es_nodes = query_nodes("(class['es_strat'] and datacenter=${::datacenter} and env=${::env} and env_num=${::env_num})")

    $es_masters = parsejson(inline_template("[<%= @query_es_nodes.map{
      |host|
        \"\\\"\" + host + \":9300\\\"\"
      }.flatten.join(', ')
      %>]"
    ))
  }

  else {
    $es_masters = $es_hosts
  }

  # Install elasticsearch and setup instances.
  class  { '::elasticsearch':
    version       => $es_version,
    init_defaults => {
      'ES_HEAP_SIZE' => $es_heap,
      'JAVA_HOME'    => "/opt/java/${java_version}/"
    },
    # Look these up again so es_masters will be included.
    instances     => $::es_strat::es_instances,
  }

  # Install plugin if defined.
  if $kopf_version {
    elasticsearch::plugin { "lmenezes/elasticsearch-kopf/${kopf_version}":
      instances  => $::hostname,
      proxy_host => 'repos.gspt.net',
      proxy_port => 3128
    }
  }

  # Install scripts if defined.
  if $es_scripts {
    create_resources(elasticsearch::script, $es_scripts)
  }

  # Setup Java in path so plugins work propperly.
  # TODO Remove this once this bug is fixed. https://github.com/elastic/puppet-elasticsearch/issues/619
  file {'/etc/sysconfig/mcollective':
    content => "export JAVA_HOME=/opt/java/${java_version}/",
    notify  => Service['mcollective'],
  }
}

Right now, the only piece that is not working is the auto-generation of the list of other Hosts that will be in the Cluster.

Tronyx
  • 1
  • 2