3

Our Perfino server crashed recently, logging since then the ERROR shown below. (There are some clues hinting to an OutOfMemory resulting in a corrupt db.)

It is suggested: 'Possible solution: use the recovery tool'. But neither the official perfino documentation nor the logs offer more instructions on how to proceed.

So here the question: how to use the recovery tool?

Stacktrace:

ERROR [collector] server: could not load transaction data
org.h2.jdbc.JdbcSQLException: File corrupted while reading record: "[495834] stream data key:64898 pos:11 remaining:0". Possible solution: use the recovery tool; SQL statement:
SELECT value FROM transaction_names WHERE id=? [90030-176]
    at org.h2.message.DbException.getJdbcSQLException(DbException.java:344)
    at org.h2.message.DbException.get(DbException.java:178)
    at org.h2.message.DbException.get(DbException.java:154)
    at org.h2.index.PageDataIndex.getPage(PageDataIndex.java:242)
    at org.h2.index.PageDataNode.getNextPage(PageDataNode.java:233)
    at org.h2.index.PageDataLeaf.getNextPage(PageDataLeaf.java:400)
    at org.h2.index.PageDataCursor.nextRow(PageDataCursor.java:95)
    at org.h2.index.PageDataCursor.next(PageDataCursor.java:53)
    at org.h2.index.IndexCursor.next(IndexCursor.java:278)
    at org.h2.table.TableFilter.next(TableFilter.java:361)
    at org.h2.command.dml.Select.queryFlat(Select.java:533)
    at org.h2.command.dml.Select.queryWithoutCache(Select.java:646)
    at org.h2.command.dml.Query.query(Query.java:323)
    at org.h2.command.dml.Query.query(Query.java:291)
    at org.h2.command.dml.Query.query(Query.java:37)
    at org.h2.command.CommandContainer.query(CommandContainer.java:91)
    at org.h2.command.Command.executeQuery(Command.java:197)
    at org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:109)
    at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:353)
    at com.perfino.a.f.b.a.a(ejt:70)
    at com.perfino.a.f.o.a(ejt:880)
    at com.perfino.a.f.o.a(ejt:928)
    at com.perfino.a.f.o.a(ejt:60)
    at com.perfino.a.f.aa.a(ejt:783)
    at com.perfino.a.f.o.a(ejt:847)
    at com.perfino.a.f.o.a(ejt:792)
    at com.perfino.a.f.o.a(ejt:787)
    at com.perfino.a.f.o.a(ejt:60)
    at com.perfino.a.f.ac.a(ejt:1011)
    at com.perfino.b.a.b(ejt:68)
    at com.perfino.b.a.c(ejt:82)
    at com.perfino.a.f.o.a(ejt:1006)
    at com.perfino.a.i.b.d.a(ejt:168)
    at com.perfino.a.i.b.d.b(ejt:155)
    at com.perfino.a.i.b.d.b(ejt:52)
    at com.perfino.a.i.b.d.a(ejt:45)
    at com.perfino.a.i.a.b.a(ejt:94)
    at com.perfino.a.c.a.b(ejt:105)
    at com.perfino.a.c.a.a(ejt:37)
    at com.perfino.a.c.c.run(ejt:57)
    at java.lang.Thread.run(Thread.java:745)
Alberto
  • 5,021
  • 4
  • 46
  • 69

2 Answers2

10

Notice: I couldn't recover my database with the procedure described below. I'm still keeping this post as reference, as the probability of a successful recovery will depend on how broken the database is, and there is no evidence that this procedure is invalid.

Perfino uses by default the H2 Database Engine as its persistence storage. H2 has a recovery tool and a run script tool to import sql statements:

# 1. Create a dump of the current database using the tool [1]
# This tool creates a 'config.h2.sql' and a 'perfino.h2.sql' db dump
cd ${PERFINO_DATA_DIR}
java -cp ${PATH_TO_H2_LIB}/h2*.jar org.h2.tools.Recover

# 2. Rename the corrupt database file to e.g. *bkp
mv perfino.h2.db perfino.h2.db.bkp

# 3. Import the dump from step 1, ignoring errors
java -cp ${PATH_TO_H2_LIB}/h2*.jar \
     org.h2.tools.RunScript \
     -url jdbc:h2:${PERFINO_DATA_DIR}/db/perfino \
     -script perfino.h2.sql -checkResults

[1]: Perfino includes a version of the h2.jar under ${PERFINO_INSTALL_DIR}/lib/common/h2.jar. You could of course download the official jar and try with it, but in my case, I could only restore the database with the jar supplied with perfino.

This failed for me with a

Exception in thread "main" org.h2.jdbc.JdbcSQLException: Feature not supported: "Restore page store recovery SQL script can only be restored to a PageStore file".

If this happens to you, try:

# 1. Delete database and mv files
cd ${PERFINO_DATA_DIR}
rm perfino.h2.db perfino.mv.db

# 2. Create a PageStore database manually
touch perfino.h2.db

# 3. try with MV_STORE=FALSE on the url [2]
java -cp ${PATH_TO_H2_LIB}/h2*.jar \
     org.h2.tools.RunScript \
     -url jdbc:h2:${PERFINO_DATA_DIR}/db/perfino;MV_STORE=FALSE \
     -checkResults \
     -continueOnError

[2]: force h2 to recreate a pagestore db instead of the new storage engine (See this thread in metabase)

Alberto
  • 5,021
  • 4
  • 46
  • 69
2

I found this article trying to repair a Confluence internal h2 database and this worked for me. Here's a shell script as a gist on my GitHub with what I did - you'll have to adjust for your environment.

AlmostGosu
  • 65
  • 6