1

In my company's CI environment we have two build servers - ubuntu server 14.04 and windows server 2008 R2 - both deployed on VMware ESX VM, with the best resources available (cpu, memory and SSD storage).

However, when we build our java code using maven, it takes about 25 minutes, whereas on each developer's local pc it takes less than 5 minutes.

We made some troubleshooting and we may have narrowed down possible causes to the integration between JVM and the actual VM (not OS). Seems like the maven process is not 'requesting' the expected disk resources it should be asking for fast performance (iops is our indication).

  1. Did any of you encounter such problems?
  2. Do you know of any special configuration needed in order to maximize JVM performance on VMware ESX VM?

Thanks!

tomper
  • 717
  • 2
  • 11
  • 25
  • Which version control do you use? How long does the checkout take? Are you using the local harddrive for build and the CI itself? Are you using a repository manager? What about unit-/integration tests? Are there any profiles on the CI system which are being automatically acitvated which locally isn't the case? ON those build machines is running only a single CI solution nothing else? Using the same JVM (JDK versions) and configurations? What do you mean by `Seems like the maven process is not 'requesting' the expected`? Are you using uptodate maven plugins? Which Maven version do you use? – khmarbaise Aug 25 '14 at 14:51
  • BTW: Is your VM configured to provide the memory staticly and not dynamicly etc.? – khmarbaise Aug 25 '14 at 14:53
  • @khmarbaise - 1. we're using git, and its performing perfecting (the maven issue is fully isolated). 2. harddrive is a remote [flash-array](http://kaminario.com/flash-array/). 3. maven profiles are the same – tomper Aug 25 '14 at 14:57
  • @khmarbaise - ... 4. maven profiles are the same. 5. the build machines are in fact jenkins slaves (not running a server app), and are used solely for the maven build. 6. jdks are the same (1.7_65 / 1.7_51). 7. we used a performance analysis tool, and we monitored the disk resources requested by maven process - which doesn't add up to the local machine's. 8. maven version is 3.0.2 and all plugins are updated. 9. and finally - VM is configured to provide memory dynamically. – tomper Aug 25 '14 at 15:04
  • Just a comment, our CI environment doesnt have this problem with virtual servers on oracle virtual box over SSH. Might a an alternative if you need one. – Mark W Aug 25 '14 at 15:04
  • You wrote the profiles are the same does that mean the same profiles are activated on local pc and on your ci? – khmarbaise Aug 25 '14 at 15:30
  • @khmarbaise - yes. same profiles are activated on local and ci. – tomper Aug 25 '14 at 15:45

1 Answers1

2

Though no clear solution was found yet and this is not a straight-forward answer, we've managed to rule out JVM performance, and made some tweaks to it which helped a bit. That was thanks to these two resources:

  1. Java Performance on ESX - A VMware community thread

  2. Java on VMware - Best Practices

... which I hope will help someone else

tomper
  • 717
  • 2
  • 11
  • 25