2

I have a Maven build job setup in Jenkins, with following notable configs :

- It is a multi-module job with 75 modules
- It is built incrementally (build only changed modules)
- Automatic artifact archiving is disabled

The build time is approx 20 mins. However, from some time the build time has drastically increased to 1 hr 15 mins. The job is still building incrementally and the current build number is about 1450. I am preserving only 3 old builds. My question is why there is such high increment in build times ?

Here are the things I have tried and observed :

Maven job creates fingerprints by default. With large number of builds (currently 1450) there may be huge fingerprint data which is being accessed during building and hence is adding to the overall build time. However, I cleaned up all the existing fingerprints and tried rebuilding the project. This had no impact on the build time (still took 1 hr to complete).

I tried doing a full build, instead of an incremental build to know if major time is being spent in identifying the modules to be build. This also did not help and the full build completed in 1 hr 20 mins.

I also tried building the job on a dedicated slave, in case the problem was due to resource availability. This had no impact on build times.

The only thing that worked was renaming the job itself. After renaming build are completing again in approx 20 mins. But I am not sure if this is an ideal solution. Also I don't understand the logic behind this approach of renaming the job.

Could someone suggest as to what is happening inside Maven build to increase the build times after a large number of builds ? I am using Jenkins v1.560 and maven project plugin v2.0. Upgrading Maven plugin or Jenkins is not an option for me now.

Update 1

As per analysis of build outputs, major build time is being spent in "Parsing POMs" (approx 30 mins) and "Metadata collection" step (approx 20 mins). Does anyone have any idea as to what exactly happens during these 2 steps ?

phoenix
  • 795
  • 1
  • 7
  • 14
  • Have you looked at _where_ that time is going? Turn on timestamping in the Jenkins logs and look for any big delays. – declension May 27 '15 at 10:06
  • @NickB "Parsing POMs" and "Metadata collection" are taking most of the time. However, I am not aware of exactly what happens underneath during these steps, so can't identify the reason for these delays. – phoenix May 27 '15 at 10:56
  • Strange! What SCM are you using? It sounds to me like this _could_ be a problem, if incremental and full builds are taking similar times (and might form part of this 'metadata collection'). Also, have you tried running this Maven build locally? – declension May 27 '15 at 11:38
  • @NickB I am using Git and Maven-3.2.1. I enabled timestamping and found that Parsing POM takes about 30 mins and Metadata collection takes 20 mins. In Jenkins I am running the build on a local disk i.e. /var/tmp/ location in slave node. When building this project locally via commandline, the full build completes in about 70 mins. – phoenix May 27 '15 at 12:01
  • Can you say how many code you have (lines of code measured by SonarQube for example) How many tests (unit and integration tests) How do you start maven in your build? Are you using a repository manager ?Are you using a a repository per build in Jenkins? Apart from that the times you have given are too long.... – khmarbaise May 28 '15 at 07:57
  • @khmarbaise There are 494983 lines of code, with about 1500 tests. The project resides in a Git repo and a fresh checkout is done on every build, however, this takes only about a minute. The build is configured to poll SCM every 5 mins. This is a Maven 2/3 job type in Jenkins and I am using maven project plugin v2.0 – phoenix May 29 '15 at 06:05
  • Does anyone have any idea about why renaming the job solves this issue ? I mean, how does job name affects build times ? Also, I started facing this problem only recently when the number of builds reached about 1400 (although I maintain a build history of only 3 old builds) Could this affect build times in any way? – phoenix May 29 '15 at 06:07
  • I have more or less the same size (460000 lines but 5600 tests) which takes less than 11 minutes overall (mvn -T 4 clean deploy) The deploy to nexus is about 1.8-2 minutes. How many modules do you have? Are you using parallelization? The `Parsing pom` to the line `Modules changed...` took about 5 seconds. The line `Scanning for projects` till the line `Reactor Build order` about 8 seconds...Where is your local repository located ? (per workspace ?) – khmarbaise May 29 '15 at 09:03
  • @khmarbaise There are 75 modules which are build incrementally. I am not using parallelization. **Parsing POMs** to **Modules changed** is taking on an average 30 mins and **Scanning for projects** to **Reactor build order** takes about 20 mins, which includes major time in **Collecting metadata** step. My workspace in located at local disk i.e. _/var/tmp/workspace_ on the Jenkins slave where build is run – phoenix May 29 '15 at 10:29
  • I am using Maven-3.2.1 to run the build. The goals invoked are **clean source:jar deploy -Pqa -U** and MAVEN_OPTS is set as **-d64 -Xms512m -Xmx3072m -XX:MaxPermSize=1024m**. I have also disabled automatic artifact archiving. – phoenix May 29 '15 at 10:31
  • Have you `Resolve Dependencies during Pom parsing` and `Process Plugins during Pom parsing` activated in Jenkins? I'm doing 320 modules and not using incremental build. Is the build running on master or slave? What about the maven repository (local to the workspace of the build job?) ? – khmarbaise May 29 '15 at 10:53
  • `Resolve Dependencies during Pom parsing` and `Process Plugins during Pom parsing` are not enabled. Build is running on slave. We have our Maven repository hosted on network using Sonatype Nexus. The local maven repo is within runtime user's home directory i.e. /u//.m2/repo and the workspace is at /var/tmp location – phoenix May 29 '15 at 11:15
  • Is the users home directory a network file system which is used by the jenkins build server ? – khmarbaise May 29 '15 at 11:17
  • Yes home directory is at NFS – phoenix May 29 '15 at 11:20
  • Ok. Than that's the problem. NFS is the worst thing you can do in a build server. Never use NFS for build servers cause it's slow and it influences your build which you can't control (network traffic makes you build slower)...Make everything on local hard disk space (I had the same issue with my build as i started with this.). Everything on NFS (Nexus, Jenkins etc.) just the deploy on nexus took on the old (12-15 minutes; NFS) now we are on a new one which has local disk space took 1.8-2 minutes)....In particular the repositories (should be local to your workspace of the build)... – khmarbaise May 29 '15 at 11:32
  • BTW: The users home directory with the maven repository is that used for several build jobs ? – khmarbaise May 29 '15 at 11:32
  • Yes users home directory is used for all maven build jobs in Jenkins. So are you suggesting that the .m2 repo should also be somewhere in /var/tmp location? That would be rather difficult as it changes Jenkins setup which may affect other jobs too :( – phoenix May 29 '15 at 11:45
  • Also, if build time is being affected by NFS then why did this issue of long build time appear only recently and not since the start. Also all the other maven jobs setup in Jenkins are running fine. This just makes me think there is something else causing problems. Moreover, as I mentioned before renaming the job is solving this problem and builds are completing in approx 20 mins! – phoenix May 29 '15 at 11:45
  • I also had the same problem check hwo many build are kept in your jenkins folder . eg : .jenkins\jobs\{job name}\builds.There should be only 5 build this will help in building the jobs fast. You can automaticallly delete these from the jenkins to. Follow the following steps : job->configure->Discard old builds->Max # of builds to keep(keep this 5). save the settings. – Praveen Jan 24 '18 at 06:12

0 Answers0