nginx reverse proxy config for Jenkins, and some 504 and slow loading problems

Question

I am battling some issues with performance of the web GUI, and having trouble tracking it down.

This is a new server, and the largest of 7, the other 6 aren't getting these issues and it's probably due to lower size and utilization. I inherited the 7 and I'm rebuilding them one by one.

First my users were getting 502s, I had to increase the buffer by a lot, they also expect to leave the console running for days/ weeks and not lose the session. Sometimes the console wouldn't respond due to no data available so I increased the timeout.

I've also configured keepalive for the max sessions I've seen in monitoring.

Then, after troubleshooting another issue where users were getting 504s on initial load, I noticed that I can't talk to 127.0.0.1:8080 from the server so that was UFW blocking traffic.

Questions:
How was any of this working if 127.0.0.1 was unreachable on 8080 ? Why does the console time out after sustained time open?
What could cause very long initial load times?

The server is taxed, and I'm going to scale it horizontally, but i'd really like to have a better understanding of this edge case.

Below is the reverse proxy config. The server has 4000 jobs and 500 agents, a 16G heap with a maximum usage of 15G this week. Garbage collection is healthy 42G of memory and 14 cores are allocated to the host.

upstream jenkins {
  keepalive 60; # keepalive connections
  server 127.0.0.1:8080; # jenkins ip and port
}

# Required for Jenkins websocket agents
map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

server {
  listen          80;       # Listen on port 80 for IPv4 requests
  listen          9090;       # Listen on port 80 for IPv4 requests

  server_name     FQDN IS HERE;  # replace with your server fqdn

  # this is the jenkins web root directory
  # (mentioned in the /etc/default/jenkins file)
  root            /var/run/jenkins/war/;

  access_log      /var/log/nginx/jenkins/access.log;
  error_log       /var/log/nginx/jenkins/error.log;

  # pass through headers from Jenkins that Nginx considers invalid
  ignore_invalid_headers off;

  location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
    # rewrite all static files into requests to the root
    # E.g /static/12345678/css/something.css will become /css/something.css
    rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
  }





  location /userContent {
    # have nginx handle all the static requests to userContent folder
    # note : This is the $JENKINS_HOME dir
    root /var/lib/jenkins/;
    if (!-f $request_filename){
      # this file does not exist, might be a directory or a /**view** url
      rewrite (.*) /$1 last;
      break;
    }
    sendfile on;
  }

  location / {
      sendfile off;
      proxy_pass         http://jenkins;
      proxy_redirect     default;
      proxy_http_version 1.1;

      # Required for Jenkins websocket agents
      proxy_set_header   Connection        $connection_upgrade;
      proxy_set_header   Upgrade           $http_upgrade;

      proxy_set_header   Host              $host;
      proxy_set_header   X-Real-IP         $remote_addr;
      proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header   X-Forwarded-Proto $scheme;
      proxy_max_temp_file_size 0;

      #this is the maximum upload size
      client_max_body_size       600m;
      client_body_buffer_size    128k;

      proxy_busy_buffers_size   512k;
      proxy_buffers   4 512k;
      proxy_buffer_size   256k;
      proxy_connect_timeout      10;
      proxy_send_timeout         432000;
      proxy_read_timeout         432000;
      proxy_buffering            on;
      proxy_request_buffering    on; # Required for HTTP CLI commands
      proxy_set_header Connection ""; # Clear for keepalive
  }

}

nginx reverse proxy config for Jenkins, and some 504 and slow loading problems

0 Answers0