Chronicle Queue - reader/tailer latency when run at same time while writing

Question

I'm setting up a market data back-testing using Chronicle Queue (CQ), reading data from a binary file then writing into a single CQ and simultaneously reading the data from that CQ and dumping the statistics. I am doing a POC to replace our existing real-time market data feed handler worker queue. While doing basic read/writes testing on Linux/SSD setup, I see reads are lagging behind writes - in fact latency is accumulating. Both Appender and Tailer are running as separate processes on same host.

Would like to know, if there is any issue in the code I am using?

Below is the code snippet -

Writer - In constructor -

    myQueue = SingleChronicleQueueBuilder.binary(queueName).build();
    myAppender = myQueue.acquireAppender();

In data callback -

    myAppender.writeDocument(myDataPacket);
    myQueue.close();

where myDataPacket is Java object wrapping the byte[] and other fields.

Tailer - In Constructor -

myQueue  = SingleChronicleQueueBuilder.binary(queueName).build();
myTailer = myQueue.createTailer();

In Read method -

    while (notLastRecord)
    {
        if(myTailer.readDocument(myDataPacket))
        {
            notLastRecord = ;
            //do stuff
        }
    }
    myQueue.close();

Any help is highly appreciated.

Thanks, Pavan

score 0 · Answer 1 · answered Nov 05 '19 at 21:26

First of all I assume by "reads are lagging behind writes - in fact latency is accumulating" you mean that for every every subsequent message, the time the message is read from the queue is further from the time the event was written to the queue.

If you see latency accumulating like that, most likely the data is produced much quicker then you can consume it which from the use case you described is very much possible - if all you need at the write side is parsing simple text line and dump it into a queue file, it's quick, but if you do some processing when you read the entry from the queue - it might be slower.

From the code it's not clear what/how much work your code is doing, and the code looks OK to me, except you probably shouldn't call queue.close() after each appender.writeDocument() call but most likely you are not doing this otherwise it would blow up.

Without seeing actual code or test case it's impossible to say more.

Chronicle Queue - reader/tailer latency when run at same time while writing

1 Answers1