7

We use Hudson as a continuous integration system to execute automated builds (nightly and based on CVS polling) of a lot of our projects.

Some projects poll CVS every 15 minutes, some others poll every 5 minutes and some poll every hour.

Every few weeks we'll get a build that fails with the following output:

FATAL: java.io.IOException: Too many open files
java.io.IOException: java.io.IOException: Too many open files
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)

The next build always worked (with 0 changes) so we always chalked it up to 2 build jobs being run at the same time and happening to have too many files open during the process.

This weekend we had a build fail Friday night (automatic nightly build) with the message and every other nightly build also failed. Somehow this triggered Hudson to continuously build every project which failed until the issue was resolved. This resulted in a build every 30 minutes or so of every project until sometime Saturday night when the issue magically disappeared.

Randyaa
  • 2,935
  • 4
  • 36
  • 49

5 Answers5

4

This is Hudson issue 715 (http://issues.hudson-ci.org/browse/HUDSON-715). The current recommendation is to set the 'maximum number of simultaneous polling threads' to keep the polling activity down.

Matt
  • 74,352
  • 26
  • 153
  • 180
Michael Donohue
  • 11,776
  • 5
  • 31
  • 44
3

See https://wiki.jenkins-ci.org/display/JENKINS/I%27m+getting+too+many+open+files+error for what we need from you to fix this kind of problem.

Renato Herebia
  • 115
  • 1
  • 8
Kohsuke Kawaguchi
  • 3,567
  • 3
  • 19
  • 21
  • We haven't had the problem in quite a long time. If it crops up again I'll be sure to send the requested information along. – Randyaa Feb 08 '10 at 14:02
  • Kohsuke - perhaps a Jenkins enhancement could be to keep an eye on file descriptor limits and self repair by running the GC then? We are running on boxes with large memory sizes (soon to be in the 96G range), so it can run for some time before trying to GC. – Danny Staple Feb 21 '11 at 12:13
  • That sounds like a great hack! Where is the RFE in http://issues.jenkins-ci.org/ ? :-) – Kohsuke Kawaguchi Mar 02 '11 at 10:20
  • link broken - Wayback Machine link - https://web.archive.org/web/20200427071509/https://wiki.jenkins.io/display/JENKINS/I'm+getting+too+many+open+files+error – Smile Sep 08 '21 at 08:04
0

I have experienced this problem with another Java application running on Debian, it went away when we downgraded to Java version 1.6.0.0. Java never closed unused connections, causing it to throw the exception.

Christofer Lundstedt
  • 11
  • 3
0

Change system limits for per-process maximum open file descriptors? As in ulimit -n for the Java process?

tzot
  • 92,761
  • 29
  • 141
  • 204
0

One of the most common problem causing "Too many open files" is to have Active Directory plugin enabled and configured in Jenkins. There are known issues with this plugin which cause enormous number of threads to show up and "Too many open files" error in logs as well. After disabling it and switching to LDAP authentication I did not experience Jenkins to hang anymore.

Szymon P
  • 1