4

We are running a couple of Puppet Master version 3.7 (Not PE) on AWS EC2 instances behind an Elastic Load Balancer (mainly for high availability).

Currently the load balancer's check is just TCP connect on port 8140. But this didn't detect that one of the instances was hung (it was still accepting the initial TCP connection but not doing anything with it).

I'm looking for way to actually send a meaningful "no-op" request to the puppet master and have it send back a result.

More specific requirements:

  1. It shouldn't trigger heavy processing (the checks happen every few seconds)
  2. It should succeed if the puppet master is able to actually handle "real request" and fail otherwise.
  3. It should be possible to execute from ELB (i.e. basically it should return "200 OK" on successful https requests on a static path.

Is there a good static path I can give the health check to use? So far I haven't found any.

030
  • 5,901
  • 13
  • 68
  • 110
Capt. Crunch
  • 839
  • 2
  • 12
  • 25

1 Answers1

4

What you could use as a check if puppet master is working correctly is to set up some kind of API query, for example:

curl -k -H "Accept: pson" https://puppetmaster:8140/production/status/no_key

Offcourse, API access won't be available by default, meaning you will have to use the SSL certificates of a signed client to be able to access the REST API. Also, all certificates won't have all the needed permissions.

This is an example how to do a full manifest compilation:

curl -k -H 'Accept: pson' \
--cert /var/lib/puppet/ssl/certs/node.example.com.pem \
--key /var/lib/puppet/ssl/private_keys/node.example.com.pem \
--cacert /var/lib/puppet/ssl/ca/ca_crt.pem \
https://puppetmaster:8140/production/status/no_key

Don't know if it's possible to integrate certificates into ELB checks, but if it's not, maybe you can play around REST API access controls and set up something along these lines in auth.conf:

# Allow ELB to access REST endpoint
path ~ ^/status/no_key$
auth off
allow_ip <ip_of_loadbalancer>

Restart puppetmaster, and try simple check:

curl -k -H "Accept: pson" https://puppetmaster:8140/production/status/no_key

This should work. Just be carefull with auth.conf not to overextend yourself!

To see all the available API keys, take a look at REST API documentation on PuppetLabs site.

Jakov Sosic
  • 5,267
  • 4
  • 24
  • 35
  • Thanks. It's not possible to upload certificates to the ELB to use as a client but it's reasonable to punch a hole through Puppet Master's authorisation to let the ELB access a specific path without requiring it. – Capt. Crunch May 07 '15 at 05:37