I first tried upgrading directly from 4.5.4 to 5.6 and got the error below, I asked our DBA to increase the DB logging level and eventually the exception stopped, but the upgrade froze during the FeedFileSources migration. In this case froze meaning several hours (10+) with no change to DB or log file (though the "web[o.s.d.v.MassUpdate] 0 files processed (0 items/sec)" continued to be printed in the log.
I've since had the DBA reduce DB logging and restore the last backup, I tried the upgrade again. This time upgrading to 4.5.7 first and upgrading all plugins to the latest 4.5.7 versions. The exception below is from this last attempt.
I could find the project that has the source code in the stack trace below and delete it, but I'm not sure it would solve the problem. We're using a postgresql DB (v9.2), what should I try next?
2016.08.08 21:25:29 INFO web[DbMigration] == FeedFileSources: migrating ================================================
2016.08.08 21:26:29 INFO web[o.s.d.v.MassUpdate] 0 files processed (0 items/sec)
2016.08.08 21:27:29 INFO web[o.s.d.v.MassUpdate] 0 files processed (0 items/sec)
2016.08.08 21:28:29 INFO web[o.s.d.v.MassUpdate] 0 files processed (0 items/sec)
2016.08.08 21:29:29 INFO web[o.s.d.v.MassUpdate] 0 files processed (0 items/sec)
2016.08.08 21:30:29 INFO web[o.s.d.v.MassUpdate] 0 files processed (0 items/sec)
2016.08.08 21:31:29 INFO web[o.s.d.v.MassUpdate] 0 files processed (0 items/sec)
2016.08.08 21:32:29 INFO web[o.s.d.v.MassUpdate] 0 files processed (0 items/sec)
2016.08.08 21:33:29 INFO web[o.s.d.v.MassUpdate] 9600 files processed (160 items/sec)
2016.08.08 21:33:33 ERROR web[o.s.s.d.m.DatabaseMigrator] Fail to execute database migration: org.sonar.db.version.v50.FeedFileSources
java.lang.IllegalStateException: Error during processing of row: [uuid=AVZtGc0P6ehXweoHtugl,uuid=AVZtGc0Q6ehXweoHtuhP,data=package com.uprr.netcontrol.dm.train.sim.rules.management.find.rules.summary_1_0.dto.request;
public enum Perspective {
PRODUCTION, SIMULATION;
public String value() {
return name();
}
public static Perspective fromValue(String v) {
return valueOf(v);
}
}
,updated_at=2015-04-17 08:05:03.932,text_value=1=560318;2=560318;3=560318;4=560318;5=560318;6=560318;7=560318;8=560318;9=560318;10=560318;11=560318;12=560318;13=560318;14=560318;15=560318;16=560318,measure_data=null,text_value=1=xprk173;2=xprk173;3=xprk173;4=xprk173;5=xprk173;6=xprk173;7=xprk173;8=xprk173;9=xprk173;10=xprk173;11=xprk173;12=xprk173;13=xprk173;14=xprk173;15=xprk173;16=xprk173,measure_data=null,text_value=1=2015-04-15T05:30:42-0500;2=2015-04-15T05:30:42-0500;3=2015-04-15T05:30:42-0500;4=2015-04-15T05:30:42-0500;5=2015-04-15T05:30:42-0500;6=2015-04-15T05:30:42-0500;7=2015-04-15T05:30:42-0500;8=2015-04-15T05:30:42-0500;9=2015-04-15T05:30:42-0500;10=2015-04-15T05:30:42-0500;11=2015-04-15T05:30:42-0500;12=2015-04-15T05:30:42-0500;13=2015-04-15T05:30:42-0500;14=2015-04-15T05:30:42-0500;15=2015-04-15T05:30:42-0500;16=2015-04-15T05:30:42-0500,measure_data=null,text_value=3=3;5=1;8=6;12=0,measure_data=null,text_value=null,measure_data=null,text_value=null,measure_data=null,text_value=null,measure_data=null,text_value=null,measure_data=null,text_value=null,measure_data=null,text_value=null,measure_data=null,text_value=null,measure_data=null,text_value=null,measure_data=null,text_value=null,measure_data=null,id=201888430]
at org.sonar.db.version.SelectImpl.newExceptionWithRowDetails(SelectImpl.java:89) ~[sonar-db-5.6.jar:na]
at org.sonar.db.version.SelectImpl.scroll(SelectImpl.java:81) ~[sonar-db-5.6.jar:na]
at org.sonar.db.version.MassUpdate.execute(MassUpdate.java:76) ~[sonar-db-5.6.jar:na]
at org.sonar.db.version.v50.FeedFileSources.execute(FeedFileSources.java:286) ~[sonar-db-5.6.jar:na]
at org.sonar.db.version.BaseDataChange.execute(BaseDataChange.java:45) ~[sonar-db-5.6.jar:na]
at org.sonar.server.db.migrations.DatabaseMigrator.executeMigration(DatabaseMigrator.java:95) ~[sonar-server-5.6.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_20]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_20]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_20]
at java.lang.reflect.Method.invoke(Method.java:483) ~[na:1.8.0_20]
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:455) [jruby-complete-1.7.9.jar:na]
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:316) [jruby-complete-1.7.9.jar:na]
... Stack trace shortened ...
at RbCallUpgrade_1691709950.callUpgrade(RbCallUpgrade_1691709950.gen:13) [jruby-complete-1.7.9.jar:na]
at org.sonar.server.ruby.PlatformRubyBridge$1.trigger(PlatformRubyBridge.java:52) [sonar-server-5.6.jar:na]
at org.sonar.server.db.migrations.PlatformDatabaseMigration.doUpgradeDb(PlatformDatabaseMigration.java:131) [sonar-server-5.6.jar:na]
at org.sonar.server.db.migrations.PlatformDatabaseMigration.doDatabaseMigration(PlatformDatabaseMigration.java:113) [sonar-server-5.6.jar:na]
at org.sonar.server.db.migrations.PlatformDatabaseMigration.access$000(PlatformDatabaseMigration.java:37) [sonar-server-5.6.jar:na]
at org.sonar.server.db.migrations.PlatformDatabaseMigration$1.run(PlatformDatabaseMigration.java:101) [sonar-server-5.6.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_20]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_20]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_20]
Caused by: java.sql.BatchUpdateException: Batch entry 191 INSERT INTO file_sources(project_uuid, file_uuid, created_at, updated_at, data, line_hashes, data_hash)VALUES ('AVZtGdtc6ehXweoHtwj4', 'AVZtGdtc6ehXweoHtwj6', 1470709529499, 1429265906896, '118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,,package com.uprr.netcontrol.bdi.outbound.documenthandler.exception;
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,,
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,,import org.slf4j.Logger;
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,,import org.slf4j.LoggerFactory;
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,,
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,,public class OutboundDocumentHandlerException extends Exception {
138289,igen585,2012-01-19T15:51:55-0600,1,,,,,,,,,,,, private static final Logger LOGGER = LoggerFactory.getLogger(OutboundDocumentHandlerException.class);
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,, private static final long serialVersionUID = 1L;
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,," public static final String DEFAULT_ERROR_KEY = ""system.error"";"
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,,
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,, public OutboundDocumentHandlerException() {
118931,xprk184,2011-10-10T07:54:05-0500,1,,,,,,,,,,,, this(DEFAULT_ERROR_KEY);
118931,xprk184,2011-10-10T07:54:05-0500,1,,,,,,,,,,,, }
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,,
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,, public OutboundDocumentHandlerException(String message) {
118931,xprk184,2011-10-10T07:54:05-0500,1,,,,,,,,,,,, super(message);
138289,igen585,2012-01-19T15:51:55-0600,1,2,1,,,,,,,,,, if (LOGGER.isErrorEnabled()){
138289,igen585,2012-01-19T15:51:55-0600,1,,,,,,,,,,,, LOGGER.error(message);
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,, }
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,,
118931,xprk184,2011-10-10T07:54:05-0500,1,,,,,,,,,,,, }
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,,
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,," public OutboundDocumentHandlerException(String message, Throwable t) {"
118931,xprk184,2011-10-10T07:54:05-0500,0,,,,,,,,,,,," super(message, t);"
138289,igen585,2012-01-19T15:51:55-0600,0,2,0,,,,,,,,,, if (LOGGER.isErrorEnabled()){
138289,igen585,2012-01-19T15:51:55-0600,0,,,,,,,,,,,, LOGGER.error(message);
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,, }
118931,xprk184,2011-10-10T07:54:05-0500,0,,,,,,,,,,,, }
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,,
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,, public OutboundDocumentHandlerException(Throwable t) {
118931,xprk184,2011-10-10T07:54:05-0500,0,,,,,,,,,,,," super(DEFAULT_ERROR_KEY, t);"
118931,xprk184,2011-10-10T07:54:05-0500,0,,,,,,,,,,,, }
118931,xprk184,2011-10-10T07:54:05-0500,,,,,,,,,,,,,}
', 'ba3bbd89d6a61e8471766df5921e6555
293f7a3f08e54359c17d5e984f721665
18d24bd6a2c2c15d3914502e2776e372
f5fbddcfb079df85f4d2407a281cdebf
f36f81f6cbcd0ad4eaf500cc094c7100
712212474a80bd328114c4c1dc66cba3
54b6fdc61bc76cf94254f77cb3da75e8
681ebf4e522395df8c779506ae70c7b7
f498d67954f51c342f0c4860f73acd92
cbb184dd8e05c9709e5dcaedaa0495cf
50c60b8d564c1374904f3625ba915e7d
7d0d74b69b4077ba7268ddf1bba275a2
0f416aef0c462782a31da800aa06c010
82ad9bf91a460c7e0a473188b7a9bdb3
cbb184dd8e05c9709e5dcaedaa0495cf
cbb184dd8e05c9709e5dcaedaa0495cf
33ea416932a5857e38856b21c2166e02
ff74d37a19fadbfb29e54b18725f0492
0f416aef0c462782a31da800aa06c010
82ad9bf91a460c7e0a473188b7a9bdb3
cbb184dd8e05c9709e5dcaedaa0495cf
cbb184dd8e05c9709e5dcaedaa0495cf
a2bb3f6e6eefb90a4a2d97a9d1deddea
b92e3d24f1d08d3b3d941ab18ef1213a
cbb184dd8e05c9709e5dcaedaa0495cf
cbb184dd8e05c9709e5dcaedaa0495cf
', '') was aborted. Call getNextException to see the cause.
at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2743) ~[postgresql-9.3-1102-jdbc41.jar:na]
at org.postgresql.core.v3.QueryExecutorImpl$1.handleError(QueryExecutorImpl.java:461) ~[postgresql-9.3-1102-jdbc41.jar:na]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1928) ~[postgresql-9.3-1102-jdbc41.jar:na]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405) ~[postgresql-9.3-1102-jdbc41.jar:na]
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2892) ~[postgresql-9.3-1102-jdbc41.jar:na]
at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297) ~[commons-dbcp-1.4.jar:1.4]
at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:297) ~[commons-dbcp-1.4.jar:1.4]
at org.sonar.db.version.UpsertImpl.addBatch(UpsertImpl.java:41) ~[sonar-db-5.6.jar:na]
at org.sonar.db.version.MassUpdate$1.handle(MassUpdate.java:80) ~[sonar-db-5.6.jar:na]
at org.sonar.db.version.SelectImpl.scroll(SelectImpl.java:78) ~[sonar-db-5.6.jar:na]
... 159 common frames omitted
2016.08.08 21:33:33 INFO web[o.s.s.d.m.PlatformDatabaseMigration] DB migration failed | time=963882ms
2016.08.08 21:33:33 ERROR web[o.s.s.d.m.PlatformDatabaseMigration] DB Migration or container restart failed. Process ended with an exception
org.jruby.exceptions.RaiseException: (StandardError) An error has occurred, all later migrations canceled:
