setup: I have Jenkins running on an ubuntu server for several months with no problems until now.
problem: For a few days now, building a job in Jenkins results in the webUI on port 8080 becoming unresponsive (ERR_CONNECTION_REFUSED or ERR_EMPTY_RESPONSE or endless loading). There is one job that on build seemingly always kills the Jenkins webUI and another job only sometimes does so.
(maybe) useful information:
the jenkins logs often include the following warnings:
2022-01-22 14:47:20.931+0000 [id=96] WARNING hudson.security.csrf.CrumbFilter#doFilter: Found invalid crumb 80e9a2cf9c3c6d86f8787587vg8f77465b9e498d818466586fb165b9430. If you are calling this URL with a script, please use the API Token instead. More information: https://www.jenkins.io/redirect/crumb-cannot-be-used-for-script
2022-01-22 14:47:20.932+0000 [id=96] WARNING hudson.security.csrf.CrumbFilter#doFilter: No valid crumb was included in request for /ajaxExecutors by <Jenkins User Id>. Returning 403.
Given these warnings, it seems to me the crumb validation fails (if so, why and how would i resolve this?), but i also suspected some memory issue somewhere, as the job that on build crashes the jenkinsUI downloads files from s3 (and cleans up afterwards). Reducing the number of downloaded file per chunk seemed to keep it from not crashing (for a short time, now its also crashing on the lower amount). So i am a little confused in which direction i should look.
Also when i ssh into to server while jenkins is down, it sometimes times out, which makes me think the whole server is overwhelmed by the execution of the jenkins job at times (maybe due an oom?)
Looking at other ppl having simular problems, i checked for phantomjs processes:
$ ps -ef | grep phantomjs | awk '{print $2}' | xargs sudo kill -9
kill: (2876): No such process
Thanks to anyone taking the time, i m completely lost with these sort of problems :D
