0

I'm having trouble with the final step in creating user certificates for use with NodeJS pg client to access a secure CockroachDB running on GKE. (it all works fine using the root user client key and cert but I don't want to use root for pg access).

The container's /cockroach-certs directory looks like this

ca.crt  client.root.crt  client.root.key

and

kubectl exec -it cockroachdb-client-secure -- ./cockroach  --certs-dir=/cockroach-certs cert list

shows

+-----------------------+------------------+-----------------+------------+--------------+-------+
|         Usage         | Certificate File |    Key File     |  Expires   |    Notes     | Error |
+-----------------------+------------------+-----------------+------------+--------------+-------+
| Certificate Authority | ca.crt           |                 | 2023/09/03 | num certs: 1 |       |
| Client                | client.root.crt  | client.root.key | 2023/09/03 | user: root   |       |
+-----------------------+------------------+-----------------+------------+--------------+-------+

I used cockroach's client-secure.yaml (https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/client-secure.yaml) - the same one used to set up the CSR for root - but changed the username to xyz (who has been added as a user to the DB). This successfully generated a CSR for xyz, which I then approved.

default.client.xyz                               8m        system:serviceaccount:default:cockroachdb   Approved,Issued

and which created the expected secret, from which I can export both a client key and cert.

default.client.xyz                Opaque                                2         9m

The problem is that cert list doesn't show client.xyz.key or client.xyz.crt and they are not in the /cockroach-certs directory. If I extract them from the default.client.xyz secret and copy them there manually, they show up in cert list but not assigned to a specific user.

The cockroach docs use "cockroach cert" to create a user, but do not show the specific process when using kubernetes. So I'm missing this last piece of the puzzle - why does client-secure.yaml work through the whole process with -user=root but miss the final step with -user=xyz, and what is the step I am missing?

.... Added more information The container claims to have written the cert files, but they are actually not there.

$ kubectl logs fidserver-csr  -c init-certs

+ /request-cert '-namespace=default' '-certs-dir=/cockroach-certs' '-type=client' '-user=fidserver' '-symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
2018/09/07 15:32:11 Looking up cert and key under secret default.client.fidserver
W0907 15:32:11.700733       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018/09/07 15:32:11 Writing cert and key to local files
wrote key file: /cockroach-certs/client.fidserver.key
wrote certificate file: /cockroach-certs/client.fidserver.crt
symlinked CA certificate file: /cockroach-certs/ca.crt -> /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

.... Updated to try again in a different certs directory - no files were actually written even though log claims they were. The symlink didn't happen either.

+ /request-cert '-namespace=default' '-certs-dir=/cockroach-client-certs' '-type=client' '-user=fidserver' '-symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
2018/09/08 09:35:56 Looking up cert and key under secret default.client.fidserver
W0908 09:35:56.160525       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018/09/08 09:35:56 Secret default.client.fidserver not found, sending CSR
Sending create request: default.client.fidserver for 
Request sent, waiting for approval. To approve, run 'kubectl certificate approve default.client.fidserver'
2018-09-08 09:36:26.718183601 +0000 UTC m=+30.564697651: waiting for 'kubectl certificate approve default.client.fidserver'
2018-09-08 09:36:56.718422397 +0000 UTC m=+60.564936446: waiting for 'kubectl certificate approve default.client.fidserver'
2018-09-08 09:37:26.718657743 +0000 UTC m=+90.565171864: waiting for 'kubectl certificate approve default.client.fidserver'
2018-09-08 09:37:56.718959817 +0000 UTC m=+120.565473905: waiting for 'kubectl certificate approve default.client.fidserver'
CSR approved, but no certificate in response. Waiting some more
request default.client.fidserver Approved at 2018-09-08 09:38:00 +0000 UTC
  reason:   KubectlApprove
  message:  This CSR was approved by kubectl certificate approve.
2018/09/08 09:38:00 Storing cert and key under secret default.client.fidserver
2018/09/08 09:38:01 Writing cert and key to local files
wrote key file: /cockroach-client-certs/client.fidserver.key
wrote certificate file: /cockroach-client-certs/client.fidserver.crt
symlinked CA certificate file: /cockroach-client-certs/ca.crt -> /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Now logged as an issue with cockroach. https://github.com/cockroachdb/cockroach/issues/29968

bruce
  • 1,408
  • 11
  • 33
  • 1
    Can you check the logs for `request-cert`? It should print out all the steps it is taking, including user and file names. – Marc Sep 07 '18 at 13:37
  • @Marc - added describe info from the pod and the csr in the question. Thank you. – bruce Sep 07 '18 at 14:09
  • I'm afraid that's not telling us much, we really need the logs from the init container as it will show the details of each step taken. – Marc Sep 07 '18 at 14:12
  • The logs for the pod are empty (both for the standard one that creates root as well as my own). Is there somewhere else I can find what's happened? And just to confirm - it is client-secure.yaml I should be using ? – bruce Sep 07 '18 at 14:50
  • 1
    Definitely `client-secure.yaml`. you could try adding a sleep or failure after the `request-cert` command so the init container keeps running. You are still using the init container but with your own username, right? The pod needs to be created from scratch when changing the init container. – Marc Sep 07 '18 at 15:08
  • @marc actually there was a log (I didn't specifically ask for logs ... -c init-container). It claims to have written the files to /cockroach-certs but it hasn't. The log looks exactly the same as the one that created the certs for root (including the comment about this might notwork). I've updated the Q with the log output. Thanks for sticking with this. – bruce Sep 07 '18 at 15:41
  • 1
    Ok, so it's definitely doing what it should. You only have one init container, right? Maybe try changing the certs-dir for both request-cert and your client – Marc Sep 07 '18 at 15:56
  • @marc tried again with a different certs directory (see Q for results). Log file looked correct but again, no actual files are written. Will log this as an issue on cockroach github as it's starting to look like it's a bug rather than me just missing something – bruce Sep 08 '18 at 10:00

1 Answers1

2

This was debugged in the github issue.

The certificate and key fetched by the request-cert init container are those for the user requested through the --user argument.

This means that the container will only have access to one set of client credentials. The volume will also not be updated with other certificates as this is only done at init time.

To request certificates for a different user, one must create a new pod including:

  • request-cert init container with --user=<desired user>
  • container making use of the client certificates for <desired user> in /cockroach-certs
Marc
  • 19,394
  • 6
  • 47
  • 51