Restarting leveldb when failed to initialize in Akka

Question

My application is designed in such a way that whenever someone tries to start it, the application will close other instances of itself. (notice that this is working as expected and should not be changed).

I am using Akka and one of my Actors is a PersistentActor that uses leveldb.

When the application starts it locks <path-to-lock>/LOCK, when it starts again the leveldb cannot lock the file so the PersistentActor cannot start.

After some inspection I found that the leveldb Actor class is LeveldbJournal and it starts under the system guardian with path akka://actor-system/system/akka.persistence.journal.leveldb.

I would like leveldb to restart itself until the file can be locked or a max retries limit has reached.

Logs:

[ERROR] [02/04/2019 15:39:25.731] [operator-actor-system-akka.actor.default-dispatcher-5] [akka://operator-actor-system/system/akka.persistence.journal.leveldb] Unable to acquire lock on '<path-to-lock>/LOCK'
akka.actor.ActorInitializationException: akka://operator-actor-system/system/akka.persistence.journal.leveldb: exception during creation
    at akka.actor.ActorInitializationException$.apply(Actor.scala:180)
    at akka.actor.ActorCell.create(ActorCell.scala:607)
    at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:461)
    at akka.actor.ActorCell.systemInvoke(ActorCell.scala:483)
    at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:282)
    at akka.dispatch.Mailbox.run(Mailbox.scala:223)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Unable to acquire lock on '<path-to-lock>/LOCK'
    at org.iq80.leveldb.impl.DbLock.<init>(DbLock.java:55)
    at org.iq80.leveldb.impl.DbImpl.<init>(DbImpl.java:167)
    at org.iq80.leveldb.impl.Iq80DBFactory.open(Iq80DBFactory.java:59)
    at akka.persistence.journal.leveldb.LeveldbStore$class.preStart(LeveldbStore.scala:178)
    at akka.persistence.journal.leveldb.LeveldbJournal.preStart(LeveldbJournal.scala:23)
    at akka.actor.Actor$class.aroundPreStart(Actor.scala:510)
    at akka.persistence.journal.leveldb.LeveldbJournal.aroundPreStart(LeveldbJournal.scala:23)
    at akka.actor.ActorCell.create(ActorCell.scala:590)
    ... 7 more

When the PersistentActor restarts:

[ERROR] [02/04/2019 15:39:57.168] [operator-actor-system-akka.actor.default-dispatcher-16] [akka://operator-actor-system/user/controller/view-manager/records-service-api-supervisor/records-service-api] Persistence failure when replaying events for persistenceId [record-service-persistence-actor]. Last known sequence number [0] (akka.persistence.RecoveryTimedOut)

Thanks,

Ido Sorozon

P.S. Versions:

Scala: 2.11.8
Akka:  2.4.19

Restarting leveldb when failed to initialize in Akka

0 Answers0