2

I'm getting "HTTP ERROR 502 Bad Gateway" when I click on a worker link in my standalone Spark UI. Looking at the master logs I can see a corresponding message...

HttpSenderOverHTTP.java:219 Generated headers (4096 bytes), chunk (-1 bytes), content (0 bytes) - HEADER_OVERFLOW/HttpGenerator@231f022d{s=START}

The network infrastructure in front of my Spark UI does indeed generate a header that is bigger than 4096 bytes, and the Spark reverse proxy is attempting to pass that to the worker UI. If I bypass that infrastructure the UI works as it should.

After digging into the Spark UI code I believe that the requestBufferSize init parameter of the Jetty ProxyServlet controls this.

Can this be increased at run-time via (say) a Java property? For example, something like...

SPARK_MASTER_OPTS=-Dorg.eclipse.jetty.proxy.ProxyServlet.requestBufferSize=8192 ...

I've tried the above without success -- I'm not familiar enough with Jetty or Servlets in general to know if that's even close to valid. Obviously I'm also looking into ways of reducing the header size but that involves systems that I have much less control over.

(Spark v3.0.2 / Jetty 9.4)

Martin Stone
  • 12,682
  • 2
  • 39
  • 53
  • were you able to solve this issue ? – Knight71 Jun 24 '22 at 16:26
  • 1
    Not in the way that I would have liked: I added a proxy to strip the headers. I've posted the details in an answer below (https://stackoverflow.com/a/72769030/44615) – Martin Stone Jun 27 '22 at 08:25
  • For us it was due to bind address , adding spark.driver.host and spark.driver.bindAddress the issue was resolved. – Knight71 Jun 28 '22 at 05:06

2 Answers2

1

Here's the workaround that I was forced to use -- Putting a proxy in front of the Spark UI that strips the headers.

I used NGINX with this in the default.conf...

server { 
  listen 8080;
  location / {
    proxy_pass http://my-spark-master:8080/;
    proxy_pass_request_headers off;
    proxy_redirect off;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}
Martin Stone
  • 12,682
  • 2
  • 39
  • 53
0

I have been fighting this 502 issue for some time now and indeed it seems to be caused by large headers from upstream proxy. I solved it by removing headers that aren't required anyways. Reviewed in browser, then remove using:

proxy_set_header Accept-Encoding "";

As an example.

Thanks for the great tip! Paul

Paul Bormans
  • 1,292
  • 16
  • 22