4

I maintain the build process for a large (> 500,000 LOC) Java project. I've just added a Sonar analysis step to the end of the nightly builds. But it takes over three hours to execute ... This isn't a severe problem (it happens overnight), but I'd like to know if I can speed it up (so that I could run it manually during work hours if desired).

Any Sonar, Hudson, Maven or JDK options I can tweak that might improve the situation?

[INFO]  -------------  Analyzing Monolith
[INFO]  Selected quality profile : Sonar way, language=java
[INFO]  Configure maven plugins...
[INFO]  Sensor SquidSensor...
[INFO]  Java AST scan...
[INFO]  Java AST scan done: 103189 ms
[INFO]  Java bytecode scan...
... (snip)
[INFO]  Java bytecode scan done: 19159 ms
[INFO]  Squid extraction...
[INFO]  Package design analysis...
... (over three hour wait here)
[INFO]  Package design analysis done: 12000771 ms
[INFO]  Squid extraction done: 12277075 ms
[INFO]  Sensor SquidSensor done: 12404793 ms

12 million milliseconds = 200 minutes. That's a long time! By comparison, the compile and test steps before the sonar step take less than 10 minutes. From what I can tell, the process is CPU-bound; a larger heap has no effect. Maybe it has to be this way because of the tangle / duplication analysis, I don't know. Of course, I know that splitting up the project is the best option! But that will take a fair amount of work; if I can tweak some configuration in the meantime, that would be nice.

Any ideas?

Zac Thompson
  • 12,401
  • 45
  • 57
  • 1
    If I get a good solution I'll make sure it gets posted here as well as into the Sonar wiki – Zac Thompson Aug 09 '10 at 17:23
  • Lucky you Zac!!... I just setup Sonar on out project... 20 hours later and the package design analysis is still going!!! That being said we are at about 1.5 million LOC :( – S73417H Feb 09 '11 at 13:37

2 Answers2

3

I walked in your shoes: on a 2million+ loc project (that should have been split into sub-projects years ago, indeed), I never saw the package design analysis to complete within 4 days of computation...

As of SONAR-2164 (Add an option to skip the quadratic "Package design analysis" phase), I have submitted a patch that would allow users to set true in their maven project file so that the package design analysis is skipped. This patch is pending approval and is currently scheduled for inclusion in v2.7.

Yannis BRES
  • 51
  • 3
1

From Freddy Mallet on the list:

"... the problem doesn't come from the DB but come from the algorithm to identify all the package dependencies to cut. ... If you manage to cut this project in several modules, then your problem will vanish."

I tested this theory by excluding a relatively large package, and sure enough it dropped dramatically. In theory the number of connections could grow quadratically with the number of packages, so this approach is probably as good as is possible with such a large codebase.

Zac Thompson
  • 12,401
  • 45
  • 57
  • Can anyone provide some input to what a "relatively large" package is? In other words, at what point did you see a performance increase? i.e. from 10,000 classes to 1,000 or maybe from 1,000 to 100? I'm going to make an explicit assumption that a package's count includes all of its subpackages. – Tony R Jan 05 '11 at 20:32
  • @Tony, yes, it was a subtree of packages. I'm afraid I can't recall the exact specifics. My guess is that it represented only a few packages (out of over 500), but perhaps 5-10% of the total number of classes (out of over 5000). But my point was that if you have 5 times more components, the runtime could be 25 times longer to analyze dependencies. If you're looking to slice off a piece, then for best results, the bigger the better (up to half). I should mention that switching to 2.4 when it came out also cut the running time more than 50%. – Zac Thompson Jan 06 '11 at 04:26