Jenkins 504 Gateway Time-out for only one single project

Question

We have a Jenkins installed on a Linux VM in GCP. The master and the slaves have more than enough CPU and memory left. This consists of multiple projects and multiple pipelines are running parallel. Recently we noticed one of projects giving 504 Gateway Time-out from time to time and being unable to go to the specific project page. The only solution we have found so far is restarting the Jenkins master. After that it will work perfectly. The Jenkins logs and the journalctl won't make any errors about this. So far, I find Jenkins is unable to give 504 errors and this is coming from somewhere else, but I am running out of ideas about how to troubleshoot this. 

I have a nginx configured in front of the Jenkins for the SSL certificates, below is the configurations.

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 768;
}

http {

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;


    include /etc/nginx/mime.types;
    default_type application/octet-stream;



    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;


    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;


    gzip on;


    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}