Can anyone point me in a direction on how to debug the following metrics-server crash? metrics-server is repeatedly crashing.
Metrics Server: 0.6.2 OS: CentOS 8 Stream K8S: 1.25 CRI: CRI-O / crun CNI: Cilium Install: Flux2 / Metrics Server Helm Chart
I0320 17:23:03.647081 1 round_trippers.go:553] GET https://192.168.4.9:10250/metrics/resource 200 OK in 3746 milliseconds
I0320 17:23:03.933618 1 secure_serving.go:352] http: TLS handshake error from 192.168.4.16:53442: write tcp 11.0.11.75:4443->192.168.4.16:53442: use of closed network connection
I0320 17:23:04.283790 1 round_trippers.go:553] GET https://192.168.4.13:10250/metrics/resource 200 OK in 7683 milliseconds
E0320 17:23:04.571445 1 writers.go:117] apiserver was unable to write a JSON response: http: Handler timeout
I0320 17:23:04.605846 1 handler.go:143] metrics-server: GET "/apis/metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1
I0320 17:23:03.949689 1 httplog.go:129] "HTTP" verb="GET" URI="/apis/metrics.k8s.io/v1beta1" latency="26.167351389s" userAgent="Go-http-client/2.0" audit-ID="414ddd42-0018-4f95-af66-7646818c2aa1" srcIP="192.168.4.9:40784" fl_authentication="15.335977207s" resp=504 statusStack=<
goroutine 1334 [running]:
k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc0000a5d90, 0x246)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/httplog/httplog.go:314 +0x8f
k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc0000a5d90, 0xd6f62d1e1f5be1f0)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/httplog/httplog.go:296 +0x25
k8s.io/apiserver/pkg/server/filters.(*baseTimeoutWriter).timeout(0xc000c0b770, 0xc00129e6e0)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/filters/timeout.go:251 +0xd8
k8s.io/apiserver/pkg/server/filters.(*timeoutHandler).ServeHTTP(0xc000382828, {0x20fca90, 0xc000c221c0}, 0xdf8475800)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/filters/timeout.go:142 +0x2ae
k8s.io/apiserver/pkg/endpoints/filters.withRequestDeadline.func1({0x20fca90, 0xc000c221c0}, 0xc00125d900)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/request_deadline.go:100 +0x494
net/http.HandlerFunc.ServeHTTP(0xff3ae7ac6fadb1e8, {0x20fca90, 0xc000c221c0}, 0xc4e93c9ce7739873)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/filters.withWaitGroup.func1({0x20fca90, 0xc000c221c0}, 0xc00125d900)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/filters/waitgroup.go:77 +0x766
net/http.HandlerFunc.ServeHTTP(0x2103138, {0x20fca90, 0xc000c221c0}, 0x20b8988)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1({0x20fca90, 0xc000c221c0}, 0xc0011a9400)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/warning.go:35 +0x2bb
net/http.HandlerFunc.ServeHTTP(0xc000382840, {0x20fca90, 0xc000c221c0}, 0xc0011a9400)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1({0x20fca90, 0xc000c221c0}, 0xc0006148c8)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/cachecontrol.go:31 +0x126
net/http.HandlerFunc.ServeHTTP(0x2103918, {0x20fca90, 0xc000c221c0}, 0x20b8988)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/httplog.withLogging.func1({0x20f06d0, 0xc0006148c8}, 0xc0010f3800)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/httplog/httplog.go:128 +0x44d
net/http.HandlerFunc.ServeHTTP(0x2103138, {0x20f06d0, 0xc0006148c8}, 0x20b8988)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1({0x20f06d0, 0xc0006148c8}, 0xc0010f3700)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/requestinfo.go:39 +0x316
net/http.HandlerFunc.ServeHTTP(0x2103138, {0x20f06d0, 0xc0006148c8}, 0xc000103a00)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.withRequestReceivedTimestampWithClock.func1({0x20f06d0, 0xc0006148c8}, 0xc0010f3600)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/request_received_time.go:38 +0x27e
net/http.HandlerFunc.ServeHTTP(0x0, {0x20f06d0, 0xc0006148c8}, 0xf5b12dac99d4f580)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithMuxAndDiscoveryComplete.func1({0x20f06d0, 0xc0006148c8}, 0xc0010f3600)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/mux_discovery_complete.go:52 +0x2c2
net/http.HandlerFunc.ServeHTTP(0x4027de4060516c2e, {0x20f06d0, 0xc0006148c8}, 0xa0c063a0d695c6a9)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1({0x20f06d0, 0xc0006148c8}, 0xc000b0b760)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/filters/wrap.go:74 +0xba
net/http.HandlerFunc.ServeHTTP(0x1cb3700, {0x20f06d0, 0xc0006148c8}, 0x8)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.withAuditID.func1({0x20f06d0, 0xc0006148c8}, 0xc001323e00)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/with_auditid.go:66 +0x40d
net/http.HandlerFunc.ServeHTTP(0x0, {0x20f06d0, 0xc0006148c8}, 0x0)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0x0, {0x20f06d0, 0xc0006148c8}, 0xc000599f80)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/handler.go:189 +0x2b
net/http.serverHandler.ServeHTTP({0x0}, {0x20f06d0, 0xc0006148c8}, 0xc001323e00)
/usr/local/go/src/net/http/server.go:2879 +0x43b
net/http.initALPNRequest.ServeHTTP({{0x2103138, 0xc000cc6210}, 0xc0006fae00, {0xc0001c68c0}}, {0x20f06d0, 0xc0006148c8}, 0xc001323e00)
/usr/local/go/src/net/http/server.go:3480 +0x245
golang.org/x/net/http2.(*serverConn).runHandler(0x1a930c0, 0xc001292200, 0x0, 0x0)
/go/pkg/mod/golang.org/x/net@v0.0.0-20220906165146-f3363e06e74c/http2/server.go:2255 +0x8c
created by golang.org/x/net/http2.(*serverConn).processHeaders
/go/pkg/mod/golang.org/x/net@v0.0.0-20220906165146-f3363e06e74c/http2/server.go:1960 +0x590
> addedInfo=<
logging error output: "{\"metadata\":{},\"status\":\"Failure\",\"message\":\"Timeout: request did not complete within the allotted timeout\",\"reason\":\"Timeout\",\"details\":{},\"code\":504}\n"
>
I0320 17:23:05.060042 1 trace.go:205] Trace[919889828]: "SerializeObject" method:GET,url:/apis/metrics.k8s.io/v1beta1,protocol:HTTP/2.0,mediaType:application/json,encoder:{"encoder":"{\"name\":\"json\",\"pretty\":\"false\",\"strict\":\"false\",\"yaml\":\"false\"}","name":"versioning"} (20-Mar-2023 17:22:35.482) (total time: 16011ms):
Trace[919889828]: ---"About to start writing response" size:292 1505ms (17:22:36.988)
Trace[919889828]: ---"Write call finished" writer:responsewriter.outerWithCloseNotifyAndFlush,size:292,firstWrite:true,err:http: Handler timeout 767ms (17:22:37.755)
Trace[919889828]: ---"About to start writing response" size:114 8631ms (17:22:46.387)
Trace[919889828]: ---"Write call finished" writer:responsewriter.outerWithCloseNotifyAndFlush,size:114,firstWrite:false,err:http: Handler timeout 842ms (17:22:47.230)
Trace[919889828]: [16.011850008s] [16.011850008s] END
I0320 17:23:05.146453 1 round_trippers.go:553] GET https://192.168.1.10:10250/metrics/resource 200 OK in 5456 milliseconds
I0320 17:23:05.187412 1 handler.go:143] metrics-server: GET "/apis/metrics.k8s.io/v1beta1" satisfied by gorestful with webservice /apis/metrics.k8s.io/v1beta1
I0320 17:23:05.326449 1 round_trippers.go:553] GET https://192.168.4.15:10250/metrics/resource 200 OK in 6509 milliseconds
I0320 17:23:05.446342 1 httplog.go:129] "HTTP" verb="GET" URI="/apis/metrics.k8s.io/v1beta1" latency="22.411647322s" userAgent="Go-http-client/2.0" audit-ID="746676f7-f8cf-49db-ba28-db142037713d" srcIP="192.168.4.9:40784" resp=503 statusStack=<
goroutine 1377 [running]:
k8s.io/apiserver/pkg/server/httplog.(*respLogger).recordStatus(0xc0010346e0, 0x641b54934e4d1a9c)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/httplog/httplog.go:314 +0x8f
k8s.io/apiserver/pkg/server/httplog.(*respLogger).WriteHeader(0xc0010346e0, 0xc001327601)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/httplog/httplog.go:296 +0x25
k8s.io/apiserver/pkg/server/filters.withWaitGroup.func1({0x20fca90, 0xc00090c3c0}, 0xc001264600)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/filters/waitgroup.go:71 +0x511
net/http.HandlerFunc.ServeHTTP(0x2103138, {0x20fca90, 0xc00090c3c0}, 0x20b8988)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithWarningRecorder.func1({0x20fca90, 0xc00090c3c0}, 0xc001264400)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/warning.go:35 +0x2bb
net/http.HandlerFunc.ServeHTTP(0xc000382840, {0x20fca90, 0xc00090c3c0}, 0xc001264400)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithCacheControl.func1({0x20fca90, 0xc00090c3c0}, 0xc000bb4050)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/cachecontrol.go:31 +0x126
net/http.HandlerFunc.ServeHTTP(0x2103918, {0x20fca90, 0xc00090c3c0}, 0x20b8988)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/httplog.withLogging.func1({0x20f06d0, 0xc000bb4050}, 0xc0002da900)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/httplog/httplog.go:128 +0x44d
net/http.HandlerFunc.ServeHTTP(0x1d5a3a0, {0x20f06d0, 0xc000bb4050}, 0x20b8988)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithRequestInfo.func1({0x20f06d0, 0xc000bb4050}, 0xc00125db00)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/requestinfo.go:39 +0x316
net/http.HandlerFunc.ServeHTTP(0x2103138, {0x20f06d0, 0xc000bb4050}, 0x0)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.withRequestReceivedTimestampWithClock.func1({0x20f06d0, 0xc000bb4050}, 0xc000bb9c00)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/request_received_time.go:38 +0x27e
net/http.HandlerFunc.ServeHTTP(0x203000, {0x20f06d0, 0xc000bb4050}, 0x0)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.WithMuxAndDiscoveryComplete.func1({0x20f06d0, 0xc000bb4050}, 0xc000bb9c00)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/mux_discovery_complete.go:52 +0x2c2
net/http.HandlerFunc.ServeHTTP(0x160, {0x20f06d0, 0xc000bb4050}, 0x0)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server/filters.withPanicRecovery.func1({0x20f06d0, 0xc000bb4050}, 0xc001327600)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/filters/wrap.go:74 +0xba
net/http.HandlerFunc.ServeHTTP(0x1cb3700, {0x20f06d0, 0xc000bb4050}, 0x8)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/endpoints/filters.withAuditID.func1({0x20f06d0, 0xc000bb4050}, 0xc0006eff00)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/endpoints/filters/with_auditid.go:66 +0x40d
net/http.HandlerFunc.ServeHTTP(0x0, {0x20f06d0, 0xc000bb4050}, 0xc00092c540)
/usr/local/go/src/net/http/server.go:2047 +0x2f
k8s.io/apiserver/pkg/server.(*APIServerHandler).ServeHTTP(0x0, {0x20f06d0, 0xc000bb4050}, 0x436cff)
/go/pkg/mod/k8s.io/apiserver@v0.23.2/pkg/server/handler.go:189 +0x2b
net/http.serverHandler.ServeHTTP({0x7f4d8217cda6}, {0x20f06d0, 0xc000bb4050}, 0xc0006eff00)
/usr/local/go/src/net/http/server.go:2879 +0x43b
net/http.initALPNRequest.ServeHTTP({{0x2103138, 0xc000cc6210}, 0xc0006fae00, {0xc0001c68c0}}, {0x20f06d0, 0xc000bb4050}, 0xc0006eff00)
/usr/local/go/src/net/http/server.go:3480 +0x245
golang.org/x/net/http2.(*serverConn).runHandler(0x6ac08e, 0xc00069e820, 0x2103138, 0xc0005ae060)
/go/pkg/mod/golang.org/x/net@v0.0.0-20220906165146-f3363e06e74c/http2/server.go:2255 +0x8c
created by golang.org/x/net/http2.(*serverConn).processHeaders
/go/pkg/mod/golang.org/x/net@v0.0.0-20220906165146-f3363e06e74c/http2/server.go:1960 +0x590
> addedInfo=<
logging error output: "{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"apiserver is shutting down\",\"reason\":\"ServiceUnavailable\",\"code\":503}\n\n"
>
E0320 17:23:05.516018 1 writers.go:117] apiserver was unable to write a JSON response: http: Handler timeout
E0320 17:23:05.749948 1 writers.go:117] apiserver was unable to write a JSON response: http: Handler timeout
I0320 17:23:05.566093 1 round_trippers.go:553] GET https://192.168.4.8:10250/metrics/resource 200 OK in 7377 milliseconds
http2: server connection error from 192.168.1.4:56880: connection error: PROTOCOL_ERROR
E0320 17:23:06.481854 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
image:
repository: registry.k8s.io/metrics-server/metrics-server
tag: v0.6.2
args:
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=30s
- --v=6
probes:
liveness:
enabled: false
readiness:
enabled: false
startup:
enabled: false
metrics:
enabled: true
serviceMonitor:
enabled: true
securityContext:
allowPrivilegeEscalation: true
privileged: true
capabilities:
add:
- CAP_NET_BIND_SERVICE
- NET_ADMIN
- NET_RAW
I've tried adding and removing the above args as well as disabling probes, as you can see above.
