1

I use k8s nginx ingress controller before my backend, having two instances. When web-cleint sends quite big request to ingress - nginx for POST/PUT/GET request always returns 400 Bad request. Btw, no another explanation or details for this error, or at least it is logged in some place where I cannot find it or identify that my problem is.

After googling I figured out the recipe to fix this: apply large_client_header_buffers with increased value: that's exactly what I did - now my buffer size is 4 256k. But no effect , I still get this error.

Pls give me any idea how to procede this problem

Eugene Shmorgun
  • 2,083
  • 12
  • 42
  • 67
  • How did you add the directive ? – XciD Apr 29 '21 at 18:55
  • Through the ConfigMap which is used by pod. I checked config file, this change is really there since I can see newly changed value. – Eugene Shmorgun Apr 29 '21 at 19:59
  • Hello @EugeneShmorgun, Please, tell me, are you using exactly this tool: https://github.com/kubernetes/ingress-nginx/ or something else? – Mikołaj Głodziak Apr 30 '21 at 10:53
  • @MikołajGłodziak yes, I do. Any value for this ? Any alternatives ? – Eugene Shmorgun Apr 30 '21 at 11:26
  • @EugeneShmorgun, Here is the information when the error occurs: "A request header field cannot exceed the size of one buffer as well, or the 400 (Bad Request) error is returned to the client." You can find more here: http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers Are you sure the request header field does not exceed the buffer size? Did you use: `large_client_header_buffers`in the config map? It should be: `large-client-header-buffers`. https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#large-client-header-buffers – Mikołaj Głodziak May 04 '21 at 13:56
  • @MikołajGłodziakc this is exactly what I did thank you, but still no success. – Eugene Shmorgun May 06 '21 at 17:21
  • @EugeneShmorgun, do you have `proxy_redirect off;` directive in your nginx config file? You can also look at the [similar topic](https://serverfault.com/questions/567456/nginxs-weird-redirect-that-includes-part-of-domain-name). – Mikołaj Głodziak May 07 '21 at 09:11

2 Answers2

2

So answer is: nginx is not guilty in described behaviour. After thoroughly investigation of log of java-app which stands behind nginx this exception was noticed

[INFO ] 2021-05-10 16:20:56.354 --- [io-10104-exec-4] org.apache.juli.logging.DirectJDKLog     : Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
 
java.lang.IllegalArgumentException: Request header is too large

And because of this detail - Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level. - it was too fleeting to be catch up during fluent observation of log.

Summing up solution was to increase SpringBoot property server.max-http-header-size to to more proper value. Default value was 8 Kb.

Eugene Shmorgun
  • 2,083
  • 12
  • 42
  • 67
1

Some additonal infromation about the problem.

The large_client_header_buffers was changed for http context over ConfigMap, server context was also changed but by simply changing the nginx.conf and reloading nginx - that didn't help either.

The problem is that the buffer chain writer buf which can be seen in the debug mode log below reaches 8k and 400 is thrown

2021/04/29 11:10:18 [debug] 805#805: *292613 http cleanup add: 0000555F4EFFC428
2021/04/29 11:10:18 [debug] 805#805: *292613 init keepalive peer
2021/04/29 11:10:18 [debug] 805#805: *292613 get keepalive peer
2021/04/29 11:10:18 [debug] 805#805: *292613 lua balancer peer, tries: 1
2021/04/29 11:10:18 [debug] 805#805: *292613 lua reset ctx
2021/04/29 11:10:18 [debug] 805#805: *292613 looking up Lua code cache with key 'balancer_by_luanhli_0f29762dfd828b8baa4d895affbc4b90'
2021/04/29 11:10:18 [debug] 805#805: *292613 stream socket 39
2021/04/29 11:10:18 [debug] 805#805: *292613 epoll add connection: fd:39 ev:80002005
2021/04/29 11:10:18 [debug] 805#805: *292613 connect to 172.18.112.41:10102, fd:39 #292619
2021/04/29 11:10:18 [debug] 805#805: *292613 http upstream connect: -2
2021/04/29 11:10:18 [debug] 805#805: *292613 posix_memalign: 0000555F4EFA1AC0:128 @16
2021/04/29 11:10:18 [debug] 805#805: *292613 event timer add: 39: 5000:5598957611
2021/04/29 11:10:18 [debug] 805#805: *292613 http finalize request: -4, "/api/assets/53f75d85-0528-434c-804f-922acb220c88?" a:1, c:2
2021/04/29 11:10:18 [debug] 805#805: *292613 http request count:2 blk:0
2021/04/29 11:10:18 [debug] 805#805: *292613 http run request: "/api/assets/53f75d85-0528-434c-804f-922acb220c88?"
2021/04/29 11:10:18 [debug] 805#805: *292613 http upstream check client, write event:1, "/api/assets/53f75d85-0528-434c-804f-922acb220c88"
2021/04/29 11:10:18 [debug] 805#805: *292613 http upstream request: "/api/assets/53f75d85-0528-434c-804f-922acb220c88?"
2021/04/29 11:10:18 [debug] 805#805: *292613 http upstream send request handler
2021/04/29 11:10:18 [debug] 805#805: *292613 http upstream send request
2021/04/29 11:10:18 [debug] 805#805: *292613 http upstream send request body
2021/04/29 11:10:18 [debug] 805#805: *292613 chain writer buf fl:1 s: 8222
2021/04/29 11:10:18 [debug] 805#805: *292613 chain writer in: 0000555F4EFB81A0
2021/04/29 11:10:18 [debug] 805#805: *292613 writev: 8222 of 8222
2021/04/29 11:10:18 [debug] 805#805: *292613 chain writer out: 0000000000000000
2021/04/29 11:10:18 [debug] 805#805: *292613 event timer del: 39: 5598957611
2021/04/29 11:10:18 [debug] 805#805: *292613 event timer add: 39: 60000:5599012614
2021/04/29 11:10:18 [debug] 805#805: *292613 http upstream request: "/api/assets/53f75d85-0528-434c-804f-922acb220c88?"
2021/04/29 11:10:18 [debug] 805#805: *292613 http upstream process header
2021/04/29 11:10:18 [debug] 805#805: *292613 malloc: 0000555F4EE529C0:4096
2021/04/29 11:10:18 [debug] 805#805: *292613 recv: eof:1, avail:-1
2021/04/29 11:10:18 [debug] 805#805: *292613 recv: fd:39 590 of 4096
2021/04/29 11:10:18 [debug] 805#805: *292613 http proxy status 400 "400 "
2021/04/29 11:10:18 [debug] 805#805: *292613 http proxy header: "Content-Type: text/html;charset=utf-8"
2021/04/29 11:10:18 [debug] 805#805: *292613 http proxy header: "Content-Language: en"
2021/04/29 11:10:18 [debug] 805#805: *292613 http proxy header: "Content-Length: 435"
2021/04/29 11:10:18 [debug] 805#805: *292613 http proxy header: "Date: Thu, 29 Apr 2021 11:10:18 GMT"
2021/04/29 11:10:18 [debug] 805#805: *292613 http proxy header: "Connection: close"
2021/04/29 11:10:18 [debug] 805#805: *292613 http proxy header done
Andrey Yaroshenko
  • 155
  • 9