Puppet: Can't manually generate cert and issue cert request over API

Question

I've spent far too much time trying to figure this out now, and I could really need a pointer. The tl;dr is that I need to manually generate a certificate on one node, and then issue a certificate request over the API.

According to the official documentation, this should be rather straight forward. Here's what I did.

Generate a certificate for myhost.foobar.local (from client)

$ puppet cert generate myhost.foobar.local

Generate a CSR from the certificate (from client)

$ openssl req -new -key /var/lib/puppet/ssl/private_keys/myhost.foobar.local.pem -subj "/CN=myhost.foobar.local" -out request.csr

Issue a certificate request to the Puppet master (from client)

The API has been opened up for remote API calls, so we can make API call from the test node. However, I only get an error back.

$ curl -k -X PUT -H "Content-Type: text/plain" --data-binary @request.csr https://puppetmaster:8140/production/certificate_request/no_key
Could not intern from s: not enough data

Other calls works just fine, such as:

$ curl -k -H "Accept: pson" https://puppetmaster:8140/production/certificate_statuses/all
[
  {
    "state" : "signed",
    "fingerprints" : {
      "default" : "5A:35:D2:19:59:C6:6E:B8:BE:64:54:FA:14:10:CE:FC:4A:C8:45:F6:DE:8E:7C:E9:2D:B0:5B:E0:5D:93:35:DD",
      "SHA256" : "5A:35:D2:19:59:C6:6E:B8:BE:64:54:FA:14:10:CE:FC:4A:C8:45:F6:DE:8E:7C:E9:2D:B0:5B:E0:5D:93:35:DD",
      "SHA1" : "04:13:AF:B9:CB:44:01:64:24:C9:E0:D6:F4:0D:60:41:52:77:EE:45",
      "SHA512" : "2C:97:11:B9:ED:38:00:1F:B0:7B:75:ED:4C:DB:B1:3E:3D:63:09:C1:38:E2:A3:4F:50:A4:FD:71:FF:55:94:C3:7A:0B:F6:D5:79:09:6D:53:39:B1:EC:C2:BF:DF:CD:9B:67:60:B9:9C:0C:82:51:E9:23:30:AA:33:AC:8B:E9:94"
    },
    "name" : "puppet.foobar.local",
    "dns_alt_names" : [
      "DNS:puppet",
      "DNS:puppet.foobar.local"
    ],
    "fingerprint" : "5A:35:D2:19:59:C6:6E:B8:BE:64:54:FA:14:10:CE:FC:4A:C8:45:F6:DE:8E:7C:E9:2D:B0:5B:E0:5D:93:35:DD"
  },
  {
    "state" : "signed",
    "fingerprints" : {
      "default" : "32:7B:B3:4E:BE:EB:66:21:E5:96:D0:7B:BA:BF:1D:FC:D5:90:E1:6F:52:6B:AB:CF:98:7E:2A:E3:48:00:A2:CF",
      "SHA256" : "32:7B:B3:4E:BE:EB:66:21:E5:96:D0:7B:BA:BF:1D:FC:D5:90:E1:6F:52:6B:AB:CF:98:7E:2A:E3:48:00:A2:CF",
      "SHA1" : "A4:17:D3:05:8A:72:BE:6C:C2:0C:FA:C4:8A:3B:6E:C4:29:90:4B:95",
      "SHA512" : "2D:C3:EE:7E:E3:39:99:C8:21:B8:97:E8:BF:FE:62:26:A8:B8:63:30:C9:F1:77:80:DB:FC:DF:B8:ED:1E:A2:6C:C2:F9:FE:5D:CA:17:D9:08:1E:EB:AA:AF:3D:99:A6:F9:3D:E6:86:A0:B3:3F:E9:EC:1C:7F:25:95:B5:D6:7C:51"
    },
    "name" : "965c252e48c3",
    "dns_alt_names" : [

    ],
    "fingerprint" : "32:7B:B3:4E:BE:EB:66:21:E5:96:D0:7B:BA:BF:1D:FC:D5:90:E1:6F:52:6B:AB:CF:98:7E:2A:E3:48:00:A2:CF"
  }
]

(formatted for readability)

I'm not sure if I'm missing something here. All other API calls appears to work fine, including signing, and DELETE/revoke of nodes. It's just the certificate request call that appears to fail. Perhaps I'm missing something obvious.

The Puppet master is running '3.7.2-1puppetlabs'.

Cross posted this issue to the Puppet Lab bug tracker https://tickets.puppetlabs.com/browse/PUP-3550 — vpetersson, Oct 28 '14 at 11:57

score 1 · Accepted Answer · answered Nov 06 '14 at 11:12

Got this sorted together with the fine folks over at Puppet Labs. The correct signing certificate request call should be:

$ curl -k -X PUT -H "Content-Type: text/plain" --data-binary @request.csr https://puppetmaster:8140/production/certificate_request/hostname.foobar.local

mc0e · Answer 2 · 2014-11-02T07:19:44.777

Firstly, I notice that in your shell snippets, your prompt is a '$'. No doubt there are puppet configurations different to mine, but I'd need to be running these commands as root. Almost certainly so on the client system, where you'd be running puppet as root. If you run the puppetmaster as something other than root, then presumably you'd run the puppet CA commands as that user.

Secondly, (and again there may be something different about your setup), I never need to run openssl commands directly. I do everything with puppet commands.

What I do is:

First, make sure your client's FQDN is as you want it to be, as this will be used in the naming of the certificate. Check with hostname -f

On client system (as root):

puppet agent --test

Assuming the key does not already exit, the above will display the fingerprint of the cert, but if you subsequently need to display it, use:

puppet agent --fingerprint --noop

On puppetmaster (as root):

puppet cert list

Check the fingerprint, then do:

puppet cert sign [client's fqdn]

If something goes wrong, then look for the various keys, certificates and certificate requests in /var/lib/puppet/ssl (on the client) and /var/lib/puppet/ssl/ca (on the master). In my experience the layout is self explanatory, and it's safe to wipe stuff out and expect it to be re-generated.

I do see that you are asking specifically about the API, but do you really need to be accessing that manually? If you do, then follow the above recipe, to generate certs, cert requests, etc, and then use openssl commands to look at the files created in the locations I've indicated, and compare to what you are generating.

If it comes down to having a look at how puppet formats the API request, what I would do is to use ltrace to catch the data passed to/from the openssl library calls. Someone more familiar with the ruby framework puppet uses might choose to insert some debug code in there instead. (ltrace is much safer on a production server though)

Yes, I need to do it manually because the way the setup is structured. I was able to get it all working using the manual commands. This is most likely an API issue, and that it's SSL'd, I can't easily sniff the traffic. Also, I did take a quick look at the code, but I'm not too familiar with the inner workings of Puppet (or Ruby tbh). — vpetersson, Nov 03 '14 at 09:38
As suggested, you can sniff the unencrypted data using ltrace. ie you sniff the library calls rather than the ethernet traffic. You won't need to know much about the code to work it out. Just run something like `ltrace puppet agent --test` and save the output to file. Then go through and look for some obvious bits of strings like 'GET' and 'POST' which you know will be in the content to be encrypted, or something more specific to the API call you're looking for, like 'production/certificate_request' — mc0e, Nov 03 '14 at 12:25
@vpetersson in your comment above you say "I was able to get it all working using the manual commands." Does that mean you answered your own question? If so, could you post it here to help someone else? — mc0e, Nov 05 '14 at 08:22
No, the whole point was that I *cannot* use the manual commands. It just proves that the system is working. I'm talking with the guys over at Puppet to try to sort it out (https://tickets.puppetlabs.com/browse/PUP-3550) — vpetersson, Nov 05 '14 at 12:01

Puppet: Can't manually generate cert and issue cert request over API

Generate a certificate for myhost.foobar.local (from client)

Generate a CSR from the certificate (from client)

Issue a certificate request to the Puppet master (from client)

2 Answers2