After configuring the aggregator, the kubeapiserver failed to start

Question

The official website of kubernetes suggests that the aggregator should be better configured different ca certificate; credential. Therefore, I followed the advice of the official website, regenerated a ca certificate, and signed the certificate to be used by the aggregator with this ca.Then I added the configuration parameter to the startup parameter of kube-apiserver according to the configuration of the official website.Then start api-server, but fail to start.The failure log is as follows:

3月 21 19:03:05 localhost.localdomain systemd[1]: Failed to start Kube-apiserver Service.
-- Subject: Unit kube-apiserver.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kube-apiserver.service has failed.
-- 
-- The result is failed.
3月 21 19:03:05 localhost.localdomain systemd[1]: kube-apiserver.service failed.
3月 21 19:03:05 localhost.localdomain kubelet[4084]: I0321 19:03:05.015767    4084 trace.go:116] Trace[1764576244]: "Reflector ListAndWatch" name:k8s.io/kubernetes
3月 21 19:03:05 localhost.localdomain kubelet[4084]: Trace[1764576244]: [14.397574036s] [14.397574036s] END
3月 21 19:03:05 localhost.localdomain kubelet[4084]: E0321 19:03:05.015796    4084 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed t
3月 21 19:03:05 localhost.localdomain kubelet[4084]: E0321 19:03:05.215925    4084 reflector.go:123] object-"kube-system"/"coredns-token-v7xr6": Failed to list *v1
3月 21 19:03:05 localhost.localdomain kubelet[4084]: I0321 19:03:05.215962    4084 trace.go:116] Trace[2021737021]: "Reflector ListAndWatch" name:object-"monitorin
3月 21 19:03:05 localhost.localdomain kubelet[4084]: Trace[2021737021]: [14.597630663s] [14.597630663s] END
3月 21 19:03:05 localhost.localdomain kubelet[4084]: E0321 19:03:05.215984    4084 reflector.go:123] object-"monitoring"/"default-token-wk7d4": Failed to list *v1.
3月 21 19:03:06 localhost.localdomain kubelet[4084]: E0321 19:03:06.000788    4084 kubelet_node_status.go:388] Error updating node status, will retry: error gettin
3月 21 19:03:07 localhost.localdomain systemd[1]: Failed to start Kube-apiserver Service.
-- Subject: Unit kube-apiserver.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kube-apiserver.service has failed.
-- 
-- The result is failed.
3月 21 19:03:07 localhost.localdomain systemd[1]: kube-apiserver.service failed.
3月 21 19:03:07 localhost.localdomain kubelet[4084]: E0321 19:03:07.215825    4084 reflector.go:123] object-"kube-system"/"coredns": Failed to list *v1.ConfigMap: 
3月 21 19:03:07 localhost.localdomain kubelet[4084]: I0321 19:03:07.215849    4084 trace.go:116] Trace[1596043133]: "Reflector ListAndWatch" name:object-"kube-syst
3月 21 19:03:07 localhost.localdomain kubelet[4084]: Trace[1596043133]: [16.600026154s] [16.600026154s] END
3月 21 19:03:07 localhost.localdomain kubelet[4084]: E0321 19:03:07.215870    4084 reflector.go:123] object-"kube-system"/"calico-kube-controllers-token-n8wt8": Fa
3月 21 19:03:07 localhost.localdomain kubelet[4084]: I0321 19:03:07.415833    4084 trace.go:116] Trace[1895303640]: "Reflector ListAndWatch" name:object-"kube-syst
3月 21 19:03:07 localhost.localdomain kubelet[4084]: Trace[1895303640]: [19.684820866s] [19.684820866s] END
3月 21 19:03:07 localhost.localdomain kubelet[4084]: E0321 19:03:07.415863    4084 reflector.go:123] object-"kube-system"/"calico-config": Failed to list *v1.Confi
3月 21 19:03:07 localhost.localdomain kubelet[4084]: E0321 19:03:07.418879    4084 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1b
 ESCOD
3月 21 19:03:05 localhost.localdomain systemd[1]: Failed to start Kube-apiserver Service.
-- Subject: Unit kube-apiserver.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kube-apiserver.service has failed.
-- 
-- The result is failed.
3月 21 19:03:05 localhost.localdomain systemd[1]: kube-apiserver.service failed.
3月 21 19:03:05 localhost.localdomain kubelet[4084]: I0321 19:03:05.015767    4084 trace.go:116] Trace[1764576244]: "Reflector ListAndWatch" name:k8s.io/kubernetes
3月 21 19:03:05 localhost.localdomain kubelet[4084]: Trace[1764576244]: [14.397574036s] [14.397574036s] END
3月 21 19:03:05 localhost.localdomain kubelet[4084]: E0321 19:03:05.015796    4084 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed t
3月 21 19:03:05 localhost.localdomain kubelet[4084]: E0321 19:03:05.215925    4084 reflector.go:123] object-"kube-system"/"coredns-token-v7xr6": Failed to list *v1
3月 21 19:03:05 localhost.localdomain kubelet[4084]: I0321 19:03:05.215962    4084 trace.go:116] Trace[2021737021]: "Reflector ListAndWatch" name:object-"monitorin
3月 21 19:03:05 localhost.localdomain kubelet[4084]: Trace[2021737021]: [14.597630663s] [14.597630663s] END
3月 21 19:03:05 localhost.localdomain kubelet[4084]: E0321 19:03:05.215984    4084 reflector.go:123] object-"monitoring"/"default-token-wk7d4": Failed to list *v1.
3月 21 19:03:06 localhost.localdomain kubelet[4084]: E0321 19:03:06.000788    4084 kubelet_node_status.go:388] Error updating node status, will retry: error gettin
3月 21 19:03:07 localhost.localdomain systemd[1]: Failed to start Kube-apiserver Service.
-- Subject: Unit kube-apiserver.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kube-apiserver.service has failed.
-- 
-- The result is failed.
3月 21 19:03:07 localhost.localdomain systemd[1]: kube-apiserver.service failed.
3月 21 19:03:07 localhost.localdomain kubelet[4084]: E0321 19:03:07.215825    4084 reflector.go:123] object-"kube-system"/"coredns": Failed to list *v1.ConfigMap: 
3月 21 19:03:07 localhost.localdomain kubelet[4084]: I0321 19:03:07.215849    4084 trace.go:116] Trace[1596043133]: "Reflector ListAndWatch" name:object-"kube-syst
3月 21 19:03:07 localhost.localdomain kubelet[4084]: Trace[1596043133]: [16.600026154s] [16.600026154s] END
3月 21 19:03:07 localhost.localdomain kubelet[4084]: E0321 19:03:07.215870    4084 reflector.go:123] object-"kube-system"/"calico-kube-controllers-token-n8wt8": Fa
3月 21 19:03:07 localhost.localdomain kubelet[4084]: I0321 19:03:07.415833    4084 trace.go:116] Trace[1895303640]: "Reflector ListAndWatch" name:object-"kube-syst
3月 21 19:03:07 localhost.localdomain kubelet[4084]: Trace[1895303640]: [19.684820866s] [19.684820866s] END
3月 21 19:03:07 localhost.localdomain kubelet[4084]: E0321 19:03:07.415863    4084 reflector.go:123] object-"kube-system"/"calico-config": Failed to list *v1.Confi
3月 21 19:03:07 localhost.localdomain kubelet[4084]: E0321 19:03:07.418879    4084 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1b
 ESCOD
3月 21 19:03:05 localhost.localdomain systemd[1]: Failed to start Kube-apiserver Service.
-- Subject: Unit kube-apiserver.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kube-apiserver.service has failed.
-- 
-- The result is failed.

All the steps I did are as follows:

step 1: Generate a certificate

mkdir -p /work/deploy/kubernetes/security/aggregatorLayer_tls 

cd /work/deploy/kubernetes/security/aggregatorLayer_tls

openssl genrsa -out ca.key 2048 

openssl req -x509 -new -nodes -key ca.key -days 10000 -out ca.pem -subj "/CN=k8s-aggregator/O=k8s-egg"

openssl genrsa -out aggregator.key 2048

openssl req -new -key aggregator.key -out aggregator.csr -subj "/O=k8s-egg/CN=aggregator"

openssl x509 -req -days 3650 -in aggregator.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out aggregator.pem

step 2：Configuration parameters

vim /etc/kubernetes/apiserver

KUBE_AGGREGATOR_ARGS="--requestheader-client-ca-file=/work/deploy/kubernetes/security/aggregatorLayer_tls/ca.pem --requestheader-allowed-names=aggregator --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --proxy-client-cert-file=/work/deploy/kubernetes/security/aggregatorLayer_tls/aggregator.pem --proxy-client-key-file=aggregator.key"

step 3: Add the boot parameters to the boot file

[root@localhost ~]# cat /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kube-apiserver Service
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

After=network.target
[Service]
Type=notify
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/apiserver
ExecStart=/usr/bin/kube-apiserver   $KUBE_LOGTOSTDERR   $KUBE_LOG_LEVEL  $KUBE_ETCD_SERVERS $KUBE_API_ADDRESS  $KUBE_API_PORT  $KUBELET_PORT $KUBE_SERVICE_ADDRESSES        $KUBE_ADMISSION_CONTROL    $KUBE_API_ARGS $KUBE_AGGREGATOR_ARGS 
Restart=always
LimitNOFILE=65536

[Install]
WantedBy=default.target

step 4: Start the kube - apiserver startup failed, the log like above

Can you provide full logs (I can see that they are being trimmed at some point). Also just to be sure can you provide the link to the documentation that you followed? Was it this [one](https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/)? — acid_fuji, Mar 20 '20 at 12:01
@acid_fuji Yes，I fellow the documentation of this[https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/] .I have modify the log in my problem (to make it more complete). And now I have a new problem that I can't create the pod successfully.(I've removed the parameters for aggregator in apiserver). I 'm not sure if the aggregator parameter I added affected something of my cluster. — Esc, Mar 21 '20 at 11:01
I still see those logs trimmed at some points. Can you paste the full ones into the question? Which kubernetes version are you using and how did you bootstrap your cluster (kubeadm, minikube)? — acid_fuji, Mar 24 '20 at 07:44

After configuring the aggregator, the kubeapiserver failed to start

0 Answers0