How to update correctly in a monitored file with CSV Source

Question

In ActivePivot 5.0, I'm monitoring a file containing my data to inject in the datastore. I want to have this single entry point, and that each change of the file updates or insert new data.

This is my configuration of the file topic:

@Bean
public CSVSource csvSource() {
    // Define source
    final CSVSource source = new CSVSource();

    final List<String> productFileColumns = Arrays.toList("a", "b", "c");
    final ICSVTopic myTopic = source.createTopic("myTopic", "path/to/my/file");
    myTopic.getParserConfiguration().setSeparator(";");
    source.addTopic(myTopic);

    // some configuration of the csv source (threads, ...)
    ...

    // Define channel factory
    final CSVMessageChannelFactory channelFactory = new CSVMessageChannelFactory(csvSource, datastore);
    ...

    // Listen some topics
    final Map<String, String> topicsToListen = new HashMap<>();
    topicsToListen.put("myTopic_Store", "myTopic");
    final Listen<IFileInfo, ILineReader> listener = new Listen<>(channelFactory, topicsToListen);
    listener.listen(csvSource);
}

And my code updating the file:

public void updateFile() {
    final Path path = Paths.get("path/to/my/file");

    // Write and truncate
    Files.write(path, "1;2;3".getBytes(), StandardOpenOption.TRUNCATE_EXISTING);

    // Add another data
    Files.write(path, "4;5;6".getBytes(), StandardOpenOption.APPEND);

    // Add another data
    Files.write(path, "7;8;9".getBytes(), StandardOpenOption.APPEND);
}

My issue is that I see multiple processings of my file by the CSV source, while I expect it to be processed only once per call to #updateFile.

Note: For now, I want to update my file using Java, not manually or with another language.

Thanks for the help.

score 0 · Answer 1 · answered Aug 20 '14 at 16:26

The CSV source listens to the modifications of your file, and processes your file each time it sees some. Whenever you are making a call to Files.write(...), you are modifying the file, the CSV source sees that and parses it. Since there are three calls to Files.write(...), it will be processed three times by the CSV source. You have two ways to implement this:

Doing a bulk write: accumulate what you need to write to the file, and only write it at the end of the function. This might however not suit your design, and use more transient memory.
Since you are rewriting the file fully anyway, you can write your data to another file, and then replace the old file with the new using Files.move with the StandardCopyOptions.ATOMIC_MOVE and StandardCopyOptions.REPLACE_EXISTING options.

How to update correctly in a monitored file with CSV Source

1 Answers1