How to restore writing to queue after failure

Question

After disk space is finish I got InternalError. Adding disk space wasn't fix problem.

Is it possible to restore and continue to persist?
May be on error I can try to recreate/close?

Creation of the queue queue = SingleChronicleQueueBuilder.binary(basePath) .build();

Writing on a single thread "TradeReactorEventPersister-1"

    ExcerptAppender appender = acquireAppender;
    if (appender == null) {
        appender = queue.acquireAppender();
        acquireAppender = appender;
    }
    appender.writeBytes(BytesStore.wrap(b));

After next exceptions:

2019-08-23 08:13:26.963 +0000 ERROR [TradeReactorEventPersister-1] LoggingUncaughtExceptionHandler - Uncaught exception a fault occurred in a recent unsafe memory access operation in compiled Java code in thread TradeReactorEventPersister-1 
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
        at net.openhft.chronicle.wire.AbstractWire.updateHeaderAssertions(AbstractWire.java:546)
        at net.openhft.chronicle.wire.AbstractWire.updateHeader(AbstractWire.java:533)
        at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreAppender.writeBytes(SingleChronicleQueueExcerpts.java:470)

2019-08-23 08:13:26.965 +0000 ERROR [TradeReactorEventPersister-1] LoggingUncaughtExceptionHandler - Uncaught exception a fault occurred in a recent unsafe memory access operation in compiled Java code in thread TradeReactorEventPersister-1 
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
        at net.openhft.chronicle.wire.AbstractWire.updateHeaderAssertions(AbstractWire.java:547)
        at net.openhft.chronicle.wire.AbstractWire.updateHeader(AbstractWire.java:533)
        at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreAppender.writeBytes(SingleChronicleQueueExcerpts.java:470)

2019-08-23 08:13:27.166 +0000 ERROR [TradeReactorEventPersister-1] LoggingUncaughtExceptionHandler - Uncaught exception a fault occurred in a recent unsafe memory access operation in compiled Java code in thread TradeReactorEventPersister-1 
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
        at net.openhft.chronicle.wire.AbstractWire.updateHeader(AbstractWire.java:511)
        at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreAppender.writeBytes(SingleChronicleQueueExcerpts.java:470)

2019-08-23 08:13:27.167 +0000 ERROR [TradeReactorEventPersister-1] LoggingUncaughtExceptionHandler - Uncaught exception you cant put a header inside a header, check that you have not nested the documents. If you are using Chronicle-Queue please ensure that you have a unique instance of the Appender per thread, in other-words you can not share appenders across threads. in thread TradeReactorEventPersister-1 
java.lang.AssertionError: you cant put a header inside a header, check that you have not nested the documents. If you are using Chronicle-Queue please ensure that you have a unique instance of the Appender per thread, in other-words you can not share appenders across threads.
        at net.openhft.chronicle.wire.AbstractWire.enterHeader(AbstractWire.java:322)
        at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreAppender.writeHeader(SingleChronicleQueueExcerpts.java:405)
        at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreAppender.writeBytes(SingleChronicleQueueExcerpts.java:463)

I couldn't persist after adding disk space. I got last exception on every trying to persist event:

2019-08-23 08:22:50.746 +0000 ERROR [TradeReactorEventPersister-1] LoggingUncaughtExceptionHandler - Uncaught exception you cant put a header inside a header, check that you have not nested the documents. If you are using Chronicle-Queue please ensure that you have a unique instance of the Appender per thread, in other-words you can not share appenders across threads. in thread TradeReactorEventPersister-1 
java.lang.AssertionError: you cant put a header inside a header, check that you have not nested the documents. If you are using Chronicle-Queue please ensure that you have a unique instance of the Appender per thread, in other-words you can not share appenders across threads.
        at net.openhft.chronicle.wire.AbstractWire.enterHeader(AbstractWire.java:322)
        at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreAppender.writeHeader(SingleChronicleQueueExcerpts.java:405)
        at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreAppender.writeBytes(SingleChronicleQueueExcerpts.java:463)

score 0 · Answer 1 · answered Nov 05 '19 at 20:58

Unfortunately chronicle queue works at the pretty low level, and it is unable to automatically repair itself after data corruption (and disk space will inevitably lead to data corruption). BTW to avoid this, Chronicle Queue will warn you if you are running low on disk space, you should've seen a warning message like: "your disk " + this.fileStore + " is almost full, warning: chronicle-queue may crash if it runs out of space."

PS you shouldn't really need to do the lazy acquire logic. You can always simply do queue.acquireAppender() - it's a cheap call which gets preexisting appender from the ThreadLocal pool, and will not create the new appender every time.

How to restore writing to queue after failure

1 Answers1