2

I am using Chronicle 4.5.27 for writing and reading market data. I have got a single writer but multiple readers. Development OS is Windows followed by Linux for Prod deployment.

How to implement following uses cases?

  1. How to start reading queues form last read position ? e.g. if reader has read 15 records from a file which has 100 records and crashed/stopped how to start reading from 16th record from next restart? Is there an inbuilt durable support in CQ?
  2. Delete files which are read by all consumers to save disk space.

For this I have implemented but seems the files are not deleted on windows due to some open issue. Is there any built support in CQ where files can be deleted only if proceeded by all interested consumers? 

public static long readMarketData(String pathForMarketDataFile, long indexFrom) {
SingleChronicleQueue queue = SingleChronicleQueueBuilder.binary(pathForMarketDataFile).rollCycle(RollCycles.MINUTELY).storeFileListener(new StoreFileListener() {
    @Override
    public void onReleased(int i, File file) {
        System.out.println("File is not in use and is ready for deletion: " + file.getName());
        try {
            file.delete();
        } catch (IOException e) {
            e.printStackTrace();
        }
        ;
        System.out.println("File deleted:  " + file.getName() );
    }

    @Override
    public void onAcquired(int cycle, File file) {
        System.out.println("File is use for reading: " + file.getName());
    }
}).build();

I have read few blogs and posts on this topic e.g.

https://vanilla-java.github.io/2016/03/29/Microservices-in-the-Chronicle-world-Part-4.html

https://groups.google.com/forum/#!topic/java-chronicle/0Nz5P-nvLgM

But still want to know if any one has implemented this use case.

Cœur
  • 37,241
  • 25
  • 195
  • 267
AmbGup
  • 731
  • 1
  • 9
  • 23

2 Answers2

2

Tracking of the consumer high-water mark is performed using MessageHistory, however, this requires that your consumers are also writing output to a chronicle queue (essentially storing the consumer read-sequence in the output queue).

Alternatively, you would need to implement your own mechanism for recording the highest sequence (index) each consumer has seen.

In terms of deleting files, there may be other processes holding open file handles to the queue files. If reader-A is no longer using queue file 15.cq4, then your code will attempt to call file.delete(), but reader-B may still have a reference to that file, stopping it from being deleted.

A more robust policy would be to have some sort of event from each reader to another service/process that is responsible for deleting the files once all readers have finished processing them.

Mark Price
  • 296
  • 1
  • 3
  • Note: Window has a issue such that even if a file is unmapped it might not be deleted until the program exits. Linux/MacOSX doesn't have this issue. – Peter Lawrey Nov 19 '17 at 23:12
  • Thanks Mark and Peter. I have implemented my own recording mechanism for last read record and for deleting files from windows I have to come out of active thread and then periodically deleting files. – AmbGup Nov 30 '17 at 16:28
  • @PeterLawrey - Can you let me know if the issue with windows has been fixed in recent versions ? – Anirudh Kashyap Jan 14 '19 at 19:28
  • @AnirudhKashyap This is an OS level issue which has been around for a decade or so now. I would be surprised if it is fixed any time soon. The simplest fix is to use Unix e.g. Centos 7.x for production. – Peter Lawrey Jan 19 '19 at 21:21
0

1) If you look at the documentation https://github.com/OpenHFT/Chronicle-Queue#restartable-tailers , Naming your tailer will take care of this functionality automatically.

You need to give unique name to each of your reading tailer, such as 'a' given in the below line of code 

 ExcerptTailer atailer = cq.createTailer("a");

The tailer store their index within the Queue itself and this index is maintained. When your reading service gets restarted, the index will get picked & tailer continue from last read position.

2) When you call file.delete(), the file does not gets deleted on windows & functionality is working fine on Linux. Windows locks files that are currently in use by the program. You cannot delete the files which your application is currently using.

There are other solutions mentioned in other threads Java 'file.delete()' Is not Deleting Specified File

Akash Gupta
  • 11
  • 1