Is there a way to avoid writing duplicate lines to a file in Java?

Question

I have a simple question:

I have some code that writes lines of data to a text file in Java. One of the method calls that writes data to the file is located within a method that is called more frequently than the data itself changes.

As a result, the text file contains numerous duplicates of each line that is written from this method. I know I can remove these duplicates from the file after the fact, but I am wondering if anyone knows of a way to avoid writing duplicate lines to a file in the first place. Thanks for reading.

The method that writes data to the file:

@Override
public void writeMessage(String location, double time, String action) {
    write(location + " " + time + " " + action);
}

And the method that calls it:

public int receiveMessage(Message m, DTNHost from) {
    int retVal = this.router.receiveMessage(m, from);

    if (retVal == MessageRouter.RCV_OK) {
        m.addNodeOnPath(this);  // add this node on the messages path
    }

    writeMessage(untranslate().toString(), SimClock.getTime(), "receiving message");

    return retVal;
}

untranslate.toString() returns a pair of xy coordinates as a string, and SimClock.getTime() returns a timestamp.

My solution was to define a String called lastMsg, initialize it as null in the constructor of the class, and then to check if the ID of the message parameter in receiveMessage() has changed before writing to the file:

public int receiveMessage(Message m, DTNHost from) {
    String currentMsg = m.getId();
    int retVal = this.router.receiveMessage(m, from);

    if (retVal == MessageRouter.RCV_OK) {
        m.addNodeOnPath(this);  // add this node on the messages path
    }
    if (!(currentMsg.equals(this.lastMsg))) {
        writeMessage(untranslate().toString(), SimClock.getTime(), "receiving message");
        this.lastMsg = currentMsg;
    }
    return retVal;
}

Add a short-demo code https://stackoverflow.com/help/minimal-reproducible-example We can only solve the problem after we understand the working of your program — Miles Morales, Sep 20 '20 at 16:05
Yes, the details of a solution would probably depend on the details of your present code, code that we cannot see. Best solution: only write to the file after the data changes and with the changed data. How to implement this? Again, it will depend on how your current program is currently wired. — DontKnowMuchBut Getting Better, Sep 20 '20 at 16:07
Do you mean duplicate *anywhere* in the file, or only duplicate *consecutive* lines? — Bohemian, Sep 20 '20 at 16:15
@MilesMorales I really was just looking for a general idea like the one provided by others. Implementing it isn't the issue. — Eric Bush, Sep 20 '20 at 16:43
@Bohemian In my case, avoiding duplicate consecutive lines is sufficient. If you have something to contribute when it comes to duplicates anywhere in the file, though, feel free to mention it. — Eric Bush, Sep 20 '20 at 16:49
You still can improve your question to help future visitors with similar problems — DontKnowMuchBut Getting Better, Sep 20 '20 at 17:03
@Bohemian I'm not sure I understand your question. The file can contain different numbers of lines depending on how long the program runs. Ideally the file would be less than 4 MB, or about 66,000 lines. — Eric Bush, Sep 21 '20 at 10:49
@EricBush that’s not much data, so you could keep a `Set` of all lines written and wrap the code that writes lines like this `if (set.add(line)) { write line to file }` — Bohemian, Sep 21 '20 at 13:58

score 1 · Accepted Answer · answered Sep 20 '20 at 16:10

Seems trivial; that method should check 'previous line written'. If the same, return;, doing nothing. If not, write it, and change the 'previous line written' field.

If you're calling this method from multiple threads, you're already in trouble (if two threads both call a method that writes 'Hello!' to a file, you could end up with a file with a line: HeHellllo!o! - you need some synchronization. Which then guarantees the field is synced too. So:

private String lastLineWritten = null;

public /* synchronized - you may need this */ void writeLog(String line) {
    if (line == null) throw new NullPointerException();
    if (line.equals(lastLineWritten)) return;
    lastLineWritten = line;
    try (BufferedWriter bw = Files.newBufferedWriter(path, StandardCharsets.UTF_8, StandardOpenOption.APPEND)) {
        bw.write(line);
    }
}

Is there a way to avoid writing duplicate lines to a file in Java?

1 Answers1