Why are all my builds failing with "Configs generator runs longer than 120 seconds"?

Question

We use TeamCity for our CI server, and have recently started using its Kotlin-based DSL to define build configurations in code and commit them to git.

Yesterday, we added a new project by creating it in Kotlin. When we pushed the repo, it appeared to work with no errors, and late in the day we got a successful build.

However, this morning, all builds in all projects - not just the new project - are failing with error messages like this: Failed to apply changes from VCS to project settings (revision c1a8904f01408c77d86794e9d276321ba11ae4d8): Configs generator runs longer than 120 seconds. Please fix the errors in VCS and commit again. That error is generated at the very beginning of the build, and the rest of the build appears to go through successfully even after it is generated.

The error message doesn't appear in Google, and I can't see anything wrong with the Kotlin locally. Have I done something wrong in my project config? Why was it working yesterday?

Answer 1

Alternatively to using a stronger machine, you can increase the 120 seconds timeout by setting the internal property teamcity.versionedSettings.configsGeneratorTimeoutSeconds to any higher seconds value by going to https://your.teamcity.instance/admin/admin.html?item=diagnostics&tab=properties, clicking Edit internal properties and then setting the value for example, to teamcity.versionedSettings.configsGeneratorTimeoutSeconds=600.

Answer 2

The answer turned out to be related to our virtualization solution - AWS EC2 - and our TeamCity server's instance size - t2.medium. T2 instances are "burstable", while other instances types like M4 instance are "fixed".

Burstable instances have a slow baseline minimum performance level, and it gains "credits" every hour that allow it to run with more CPU resources than the baseline. When instances are new, or if they are not constantly using CPU, they will generally have enough CPU credits to run quickly at the higher performance level, but when those credits are depleted, the instance will run very slowly at the baseline performance level.

This is what happened to us. Apparently, the baseline performance level was so slow that it couldn't generate TeamCity's config from Kotlin in two minutes for our very modest needs (7 build configurations across 3 projects) at the baseline performance level, so all the builds were failing.

Once we changed our TeamCity server to use a higher performing instance size, TeamCity could generate its config from Kotlin quickly again, and our problem was resolved.