1

We are upgrading our SonarQube from 4.5.X to 5.6.X, compute engine is a new concept and on the paper is pretty shinny. But when put to work, it does not perform much! help me get through if you had made it

I choose a test projects that was scanned under 4 minutes with SonarQube 4.5.4. Now the initial scan taking place with (gradle) scanner takes 3 minutes. And CE background tasks runs forever, out of the test projects, first one completed in 183 minutes, other in 263 minutes, third one is still running.

Obviously I got digging and saw this stackoverflow question - Sonarqube background tasks running forever and increased the memory as quoted as answer, but its is not working for me. 

sonar.ce.javaOpts=-Xmx4096m -Xms512m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true
sonar.ce.workerCount=5

Though worker count is 5, I'm not running parallel tasks now. From the completed job's logs I'm able to see some alarming time info!

2018.02.13 16:56:07 INFO  [o.s.s.c.s.ComputationStepExecutor] Compute complexity measures | time=335ms
2018.02.13 16:56:07 INFO  [o.s.s.c.s.ComputationStepExecutor] Load measure computers | time=6ms
2018.02.13 21:18:34 INFO  [o.s.s.c.s.ExecuteVisitorsStep]   Execution time for each component visitor:
2018.02.13 21:18:34 INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - LoadComponentUuidsHavingOpenIssuesVisitor | time=2483ms
2018.02.13 21:18:34 INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - IntegrateIssuesVisitor | time=15743879ms
2018.02.13 21:18:34 INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - CloseIssuesOnRemovedComponentsVisitor | time=0ms
2018.02.13 21:18:34 INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - QualityModelMeasuresVisitor | time=143ms
2018.02.13 21:18:34 INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - NewQualityModelMeasuresVisitor | time=73ms
2018.02.13 21:18:34 INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - LastCommitVisitor | time=4ms
2018.02.13 21:18:34 INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - MeasureComputersVisitor | time=25ms
2018.02.13 21:18:34 INFO  [o.s.s.c.s.ComputationStepExecutor] Execute component visitors | time=15746700ms
2018.02.13 21:18:37 INFO  [o.s.s.c.s.ComputationStepExecutor] Compute measure variations | time=3290ms
2018.02.13 21:18:37 INFO  [o.s.s.c.s.ComputationStepExecutor] Compute Quality Gate measures | time=54ms
2018.02.13 21:18:37 INFO  [o.s.s.c.s.ComputationStepExecutor] Compute Quality profile measures | time=25ms
2018.02.13 21:18:38 INFO  [o.s.s.c.s.ComputationStepExecutor] Generate Quality profile events | time=33ms
2018.02.13 21:18:38 INFO  [o.s.s.c.s.ComputationStepExecutor] Generate Quality gate events | time=6ms

Execute component visitor, particularly IntegrateIssuesVisitor takes hell a lot of time, 262 minutes out of total 263 minutes required for execution. How to avoid this?

I don't see any cross project analysis disable option, is it related to that?

Additional information: I use Oracle DB

[Update] Crossed out curse on CE. My bad, CE is awesome! Now some projects are analysed and published in 3 seconds!!

Shiva
  • 717
  • 9
  • 22
  • 2
    First, 5 workers is likely too many for your infrastructure. See the [guidance in the docs](https://docs.sonarqube.org/display/PLUG/Managing+Compute+Engine+Performance). Second, 6.7.1 is the current LTS. If you're migrating, you should jump all the way; there have been many enhancements and improvements between the two versions (including performance enhancements!) – G. Ann - SonarSource Team Feb 16 '18 at 15:02
  • Thanks for the comments @G.Ann-SonarSourceTeam but the worker count is just there, I'm not even using 2 threads. Also I have allocated 4gb of Heap space. CPU is 99% idle and 11gb of 15gb usable RAM is free! consider this as baby step, we are planning 6.7.1 but only after successfully migrating to 5.6.7 – Shiva Feb 16 '18 at 15:16
  • It would be great if I can get insight into `IntegrateIssuesVisitor | time=15743879ms` – Shiva Feb 16 '18 at 15:18
  • With same java options, reduced **worker count to 1**; still time the run has not come down! I just let the long running scan complete over weekend and it took 13 hours!!! – Shiva Feb 20 '18 at 06:51
  • 1
    It sounds like the box hosting your SonarQube server is just under-powered. The introduction of the Compute Engine moved the heavy lifting from the analysis box to the SQ box. – G. Ann - SonarSource Team Feb 20 '18 at 12:13

1 Answers1

2

Solved!

It had to do with Database, as discussions on SonarQube Google groups pointed out.

From https://docs.sonarqube.org/display/PLUG/Managing+Compute+Engine+Performance, It is clear that Compute Engine performance are tied to,

  • DB
  • disk I/O
  • network
  • heap
  • CPU

Our memory consumption and CPU usage were always minimal and so straight away we ruled out last two. But we didn't know what was wrong yet, if anything CE logs are useless in every way since log level is not applied to CE logs, you are stuff with only INFO no DEBUG or TRACE levels. Still we took a hint, 

INFO  [o.s.s.c.s.ExecuteVisitorsStep]   Execution time for each component visitor:
INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - LoadComponentUuidsHavingOpenIssuesVisitor | time=58ms
INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - IntegrateIssuesVisitor | time=41147606ms
INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - CloseIssuesOnRemovedComponentsVisitor | time=1ms
INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - QualityModelMeasuresVisitor | time=206ms
INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - NewQualityModelMeasuresVisitor | time=119ms
INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - LastCommitVisitor | time=6ms
INFO  [o.s.s.c.s.ExecuteVisitorsStep]   - MeasureComputersVisitor | time=29336ms

IntegrateIssuesVisitor was taking outrageous amount of time to execute, the log was only that informative and the why part was still mystery. So we decided to dig the code, thank god SonarQube is open source but soon enough or DBA team rang us up why our query is behaving so badly. And it stuck us, indexes we Analyzed the transaction tables of Sonar and we are good now.

tl;dr : Keep your table statistics updated so optimization are always working in your favor. Analyze tables.

Shiva
  • 717
  • 9
  • 22