0

I've been trying to migrate from an outdated sonar installation to a newer SonarQube, version 5.1.2. The application is running on a Solaris platform and supported by an Oracle database.

Most everything works fine, except for the source code not being shown for any source file, in any context.

Any request to */api/sources/lines?uuid=...&from=1&to=1000 results in a 404, with a json object containing the following message:

"File '...' has no sources"

I've tried running the analysis from our Hudson server and from maven, with the same results. After having browsed the web for a few hours and found no question with the same symptoms, I'm turning to SO for help. In all the topics I've read, either the analysis doesn't run (e.g. Unable to run analysis on SonarQube 5.1) or the code is shown for some issues and not for others (e.g. Sonarqube Javascript Coverage doesn't show source files).

Out of sheer lassitude, I've tried installing the exact same version of SonarQube (5.1.2, same zip file as extracted on solaris), with the same configuration on my local machine (running Linux). The only differences are:

  • the supporting OS (Solaris / Linux)
  • the database (Oracle / H2)
  • the JDK (although both are oracle's 1.8.0_45)

To my dismay, it works perfectly fine.

I've enabled DEBUG logs on the server and haven't been able to get any further details on how / why the elasticsearch index isn't finding the source for our projects.

Any information regarding how I can investigate further on this issue would be welcome.

Community
  • 1
  • 1
Thomas
  • 705
  • 5
  • 13
  • From which version do you upgrade ? – Simon Brandhof Oct 10 '15 at 06:29
  • You should check your account has the permissions "See Source Code" and "Browse" (see console Settings -> Security -> Project Permissions) – Simon Brandhof Oct 10 '15 at 06:33
  • I'll have to get back to you with the exact version I upgraded from, but I believe it was 3.2.1... – Thomas Oct 11 '15 at 09:22
  • The permissions are fine. Browse and "See source code" are granted to "anyone"... The error I get a 404, with the message "File ... has no source", rather than a 403, which I'd expect if the permissions were wrong. (Traced the Ajax call in chrome's developer window) – Thomas Oct 11 '15 at 09:28
  • In fact the status is 404 even if the file exists but permissions are not granted. That prevents from giving informations about resources that user is not supposed to know. – Simon Brandhof Oct 11 '15 at 12:33
  • Do you confirm that the sources are missing in both web service and webapp ? Are these sources correctly taken into account of measures (for example metrics on numbers of files or lines of code) ? – Simon Brandhof Oct 12 '15 at 07:17
  • Indeed. My metrics are fine, the issues are detected, only the source is missing. – Thomas Oct 12 '15 at 07:52
  • what about new projects ? Do they have the same pb ? – Simon Brandhof Oct 12 '15 at 12:21
  • Yup. At least I've run the analysis changing the "branch" property, which results in its being shown as a new project. The sources were missing there too. – Thomas Oct 12 '15 at 14:19
  • 1
    There are too many versions between 3.2 and 5.1. Investigation of the pb is too painful. You should try to upgrade to the Long Term Support version (4.5.x) before upgrading to 5.1. – Simon Brandhof Oct 12 '15 at 15:35
  • Sorry for the much delayed answer, I've had a busy couple of weeks on unrelated matters. I've restored the previous database, installed and configured the 4.5.6 LTS version, and the source does appear correctly. I'll try and get a backup done at this point, and upgrade from there back to 5.1.2, and I'll keep you posted. Thanks again for the help! – Thomas Oct 28 '15 at 13:04
  • @SimonBrandhof-SonarSource Done just that (over the course of several days and with some minor hiccups along the way) and it seems to have worked! Thanks a bunch. – Thomas Nov 03 '15 at 13:20

2 Answers2

2

Projects must be analyzed again when upgrading from (very) old versions such as 3.2. That should bring display of sources back.

Simon Brandhof
  • 5,137
  • 1
  • 21
  • 28
  • I already ran the analysis again, several times. Using both the maven runner and the Hudson plugin. – Thomas Oct 11 '15 at 16:03
  • Thanks for the help though! It would have been a very good reason indeed. The sources moved from the database to the elastic search index, right? – Thomas Oct 11 '15 at 16:04
0

I've been able to work around my issue following Simon's advice.

I've

  1. restored the backup we had from the database before the upgrade from 3.2.1 to 5.1.2
  2. performed an upgrade to the 4.5.6 LTS version
  3. run a few analysis over several days on the 4.5.6 version, making sure everything worked as intended
  4. created a backup of the database for the 4.5.6 version
  5. performed the upgrade to version 5.1.2

During the migration to 5.1.2, the IndexSynchronizer ran into some trouble (due to elasticsearch apparently overloading the host's CPU):

 2015.11.03 13:21:23 INFO  web[o.s.s.s.IndexSynchronizer] Index source lines
 2015.11.03 13:22:10 INFO   es[o.e.monitor.jvm]  [sonar-1446552525578] [gc][young][766][38] duration [784ms], collections [1]/[1s], total [784ms]/[6.6s], memory [194mb]->[126.7mb]/[989.8mb], all_pools {[young] [58.6mb]->[2.1kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old] [126.8mb]->[118.2mb]/[682.6mb]}
 2015.11.03 13:22:23 INFO  web[o.s.s.es.BulkIndexer] 110525 requests processed (1842 items/sec)
 2015.11.03 13:22:30 WARN   es[o.e.monitor.jvm]  [sonar-1446552525578] [gc][young][784][51] duration [2.1s], collections [2]/[2.9s], total [2.1s]/[10.7s], memory [175.6mb]->[148.3mb]/[989.8mb], all_pools {[young] [42.2mb]->[37.3kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old] [124.9mb]->[139.7mb]/[682.6mb]}
 2015.11.03 13:23:23 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (269 items/sec)
 2015.11.03 13:24:23 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:25:23 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:26:23 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:27:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:28:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:29:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:30:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:31:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:32:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:33:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:34:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:35:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:36:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:37:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:38:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:39:24 INFO  web[o.s.s.es.BulkIndexer] 126712 requests processed (0 items/sec)
 2015.11.03 13:40:07 INFO  web[o.e.client.transport] [sonar-1446552525578] failed to get node info for [#transport#-1][localhost][inet[/127.0.0.1:9001]], disconnecting...
 org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[/127.0.0.1:9001]][cluster:monitor/nodes/info] request_id [3417] timed out after [5834ms]
    at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366) ~[elasticsearch-1.4.4.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
 2015.11.03 13:40:07 WARN   es[o.e.monitor.jvm]  [sonar-1446552525578] [gc][young][788][54] duration [17.5m], collections [1]/[17.5m], total [17.5m]/[17.7m], memory [101mb]->[104.3mb]/[989.8mb], all_pools {[young] [7.8mb]->[380.8kb]/[273mb]}{[survivor] [8.5mb]->[8.5mb]/[34.1mb]}{[old] [84.7mb]->[95.5mb]/[682.6mb]}
 Wrapper Process has not received any CPU time for 978 seconds.  Extending timeouts.
 2015.11.03 13:40:07 ERROR web[o.s.s.ui.JRubyFacade] Fail to upgrade database
 java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Fail to execute ES search request '{"size":100,"query":{"filtered":{"query":{"match_all":{}},"filter":{"bool":{"must":[{"term":{"fileUuid":"3d674506-5a64-4b1e-a849-2c051e736e55","_cache":false}},{"range":{"line":{"from":250,"to":null,"include_lower":false,"include_upper":true},"_cache":false}}],"_cache":false}}}},"_source":false}' on indices '[sourcelines]' on types '[sourceline]'
    com.google.common.base.Throwables.propagate(Throwables.java:156)
    org.sonar.server.es.BaseIndexer.index(BaseIndexer.java:77)
    org.sonar.server.search.IndexSynchronizer.execute(IndexSynchronizer.java:90)

After that, it simply hung and wouldn't proceed with the upgrade nor show the index page (both / redirected to /maintenance and /setup showed an error message). After stopping and restarting sonar, it eventually finished migrating the source files index, and from there on it seems to work as expected.

Thomas
  • 705
  • 5
  • 13