0

I have a running GKE cluster with cockroachDB active. It's been running for quite a while and I don't want to reinitialize it from scratch - it uses the (almost) standard cockroachDB supplied yaml file to start. I need to change a switch in the exec line to modify the logging level -- currently it's set to the below (but that is logging all information messages as well as errors)

exec /cockroach/cockroach start --logtostderr --insecure --advertise-host $(hostname -f) --http-host 0.0.0.0 --join cockroachdb-0.cockroachdb,cockroachdb-1.cockroachdb,cockroa
chdb-2.cockroachdb --cache 25% --max-sql-memory 25%"

How do I do this without completely stopping the DB?

bruce
  • 1,408
  • 11
  • 33

1 Answers1

2

Kubernetes allows you to update StatefulSets in a rolling manner, such that only one pod is brought down at a time.

The simplest way to make changes is to run kubectl edit statefulset cockroachdb. This will open up a text editor in which you can make the desired change to the command, then save and exit. After that, Kubernetes should handle replacing the pods one-by-one with new pods that use the new command.

For more information:

Alex Robinson
  • 12,633
  • 2
  • 38
  • 55
  • Alex - I tried this and noticed that it is indeed already set to --logtostderr=ERROR already. The problem I'm trying to solve is a huge amount of logging of informational messages to Stackdriver as described here https://stackoverflow.com/questions/51304889/cockroachdb-kubernetes-logging-lots-of-errors-on-stackdriver/51305673#51305673 . How can i avoid this ? – bruce Jul 13 '18 at 15:56
  • Just to be clear, by default the Kubernetes config file sets `--logtostderr`, but not `--logtostderr=ERROR`. If the level isn't specified, it's implicitly interpreted as `INFO`. But assuming that it is indeed `--logtostderr=ERROR`, it's possible you'll have to manually delete the pods one-by-one so that they can be recreated with the desired command line. I wouldn't expect that to be necessary, but you could be (1) using an old version of kubernetes (2) using an old version of the config file, (3) have modified the config file, or (4) maybe `command` changes don't trigger a rolling update? – Alex Robinson Jul 15 '18 at 04:37
  • I am indeed still using an older version (pre 2.0) of cockroachDB. Modifying the yaml file to add logtostderr=ERROR, reapplying it, and deleting the pods fixed it. Thanks so much for your help and patience. The default of logtostderr=INFO was the root of the problem. – bruce Jul 15 '18 at 10:07