0

We're using both TeamCity and Gerrit. For some connections to Gerrit TeamCity does not close the connection. This does not happen all the time, but frequently enough to cause problems. After a couple days the max allowed concurrent connections of 64 is reached and teamcity is no longer able to connect at all.

We've enabled a timeOut in TeamCity of one minute after which the build is marked as failed and the connection should be closed.

I've tried to close the connections manually using 'ssh -p 29418 $GIT_HOST gerrit close-connection $ID' but this did not work. Also setting the sshd.timeOut did not make any difference.

Is there another way to forcibly close connections and what could explain the issue of not closing the connection the first place?

Note that we use other tools like Jenkins as well. There is a continuous flow of connections all the time and we only have issues with TeamCity when it comes to closing the connection.

JohanKees
  • 669
  • 1
  • 12
  • 22
  • How are you actually using TeamCity? Some plugin involved or it's just a sequence of commands executed by Command Line runner? – cyberskunk Mar 28 '16 at 10:54
  • We're using the git plugin. – JohanKees Mar 28 '16 at 11:20
  • What makes you think that TeamCity is responsible for those connections? Do you have any warnings/errors in logs (have you tried turning debug mode on)? – cyberskunk Mar 28 '16 at 13:28
  • Not saying it's either Teamcity or Gerrit at the moment. Can be either of them. Just noticed that TeamCity is the only client impacted by this. The error we see is: [2016-03-20 02:01:12,470] WARN - jetbrains.buildServer.SERVER - Error while starting build {id=193365}: jetbrains.buildServer.serverSide.impl.BuildChangesCollectException: Failed to collect changes, error: List remote refs failed: com.jcraft.jsch.JSchException: channel request: timeout (enable debug to see stacktrace) – JohanKees Mar 29 '16 at 08:57
  • Then the logs from the other side might be quite useful to understand why there's connection timeout there. – cyberskunk Mar 29 '16 at 09:37
  • Yeah, working on that. So far no luck. Every time I monitor the logs after increasing the log level to TRACE the issue disappears. Given the log growth I cannot have it running over night. And when I reduce the log level... after some time the issue re-appears again. – JohanKees Mar 30 '16 at 15:55
  • Do you use the commit-status-publisher plugin in TeamCity? – neverov Mar 31 '16 at 07:22
  • What's your TeamCity version, plain 9.0 or some update of it? – neverov Mar 31 '16 at 07:23
  • I was able to reproduce the issue somewhat by cancelling a clone operation while it was "Fetching sources". An error is shown: "fatal: The remote end hung up unexpectedly" and a stale connection is left on the server – JohanKees Apr 06 '16 at 12:11
  • We're on Teamcity 9.1.6 and use the git and gerrit plugins – JohanKees Apr 06 '16 at 12:38

0 Answers0