How can I restore the database when mesh is not starting up any more?

Question

Due to low memory our mesh instance crashed and left the database in an inconsistent state. Mesh is not starting up anymore and I have no chance to use the restore end points. Is there an alternative way to restore the database?

Here is the error output:

SEVERE: $ANSI{green {db=storage}} Exception during index 'e.has_branch_inout' deletion
com.orientechnologies.orient.core.exception.OPageIsBrokenException: Following files and pages are detected to be broken ['e.has_field_container_branch_type_lang.hib' :86;], storage is switched to 'read only' mode. Any modification operations are prohibited. To restore database and make it fully operational you may export and import database to and from JSON.
DB name="storage"
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkLowDiskSpaceRequestsAndReadOnlyConditions(OAbstractPaginatedStorage.java:5143)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.deleteIndexEngine(OAbstractPaginatedStorage.java:2113)
        at com.orientechnologies.orient.core.index.OIndexAbstract.create(OIndexAbstract.java:259)
        at com.orientechnologies.orient.core.index.OIndexMultiValues.create(OIndexMultiValues.java:258)
        at com.orientechnologies.orient.core.index.OIndexMultiValues.create(OIndexMultiValues.java:44)
        at com.orientechnologies.orient.core.index.OIndexManagerShared$RecreateIndexesTask.createAutomaticIndex(OIndexManagerShared.java:600)
        at com.orientechnologies.orient.core.index.OIndexManagerShared$RecreateIndexesTask.recreateIndex(OIndexManagerShared.java:586)
        at com.orientechnologies.orient.core.index.OIndexManagerShared$RecreateIndexesTask.recreateIndexes(OIndexManagerShared.java:534)
        at com.orientechnologies.orient.core.index.OIndexManagerShared$RecreateIndexesTask.run(OIndexManagerShared.java:515)
        at java.lang.Thread.run(Thread.java:745)

Thank you for help!

I have not yet seen this issue before - If you want you could provide me with the DB and I can take a look at it. — Jotschi, Aug 31 '18 at 10:45

score 1 · Answer 1 · answered Sep 04 '18 at 07:55

We managed to recover the database by using the OrientDB shell.

First we deleted all the *.hib files
Started the OrientDB shell via docker
Then we exported the old data
Created a new database
Finally we imported the old data to the new database and started mesh

score 0 · Answer 2 · answered Aug 31 '18 at 10:44

You could try to delete all the *.hib files in your storage directory and start Mesh again. (Create a backup up-front)

Otherwise you would unfortunately need to download OrientDB 2.2.x, add the mesh db in the OrientDB installation and invoke the import/export:

https://orientdb.com/docs/2.2.x/Export-Format.html

If the REST API of Gentics Mesh is reachable you could also try to invoke the backup and restore endpoint. This could be done via the mesh-cli:

mesh admin backup
mesh admin restore

How can I restore the database when mesh is not starting up any more?

2 Answers2