Only reading additional content in a file

Question

I'm working on a project which requires reading a log file, The log file can get massive, so I only want to read additional data added to the file and not the old data to avoid potential performance issues.

I tried fs.createReadStream but it didn't seem to work:

code:

let stream = fs.createReadStream(`/path/`, { autoClose: true, encoding: "utf-8" });

stream.on('ready', () => {
    console.log("Loaded log file!");
})

stream.on("data", chunk => {

    // Data stream

});

stream.on("end", () => {
    stream.pause();

    console.log("ended");

    // Going thru the data

    setTimeout(() => {
        stream.resume();
        stream.read();
    }, 10000);

});

With that code, the "end" event only triggers once, although the stream is set to not close automatically, and no additional data is going thru.

You could save the line you last ended on and continue from there — KyleRifqi, Jan 28 '22 at 12:48
@KyleRifqi How can I do that, cause as far as I know, if I'm reading the file again, it's going to use a lot of resources since I'm reading the entire file. which is not really ideal when you're reading it every 2 seconds. So if there's a way to start from a certain chunk using ReadStream that'd be exactly what i need! — whydoihavetochoseaname, Jan 29 '22 at 01:48
after googling around it doesnt look like theres a way to specify a line to start on but there is the `start` options that you can use to skip over a number of bytes, so you could count the number of bytes youve read in a variable and use that as the start everytime you read the file again. this is just theoretical though, i have not tried this before. — KyleRifqi, Jan 29 '22 at 05:54

score 1 · Accepted Answer · edited Feb 02 '22 at 04:54

After messing a bit with the readStream and thanks to @KyleRifqi I was able to come up with this solution, storing the last amount of bytes and updating it everytime the stream ends.

let lastBytes = 0;

let text = "";

let stream = fs.createReadStream(`/path/`, { autoClose: true, encoding: "utf-8", start: lastBytes });

stream.on('ready', () => {
    console.log("Loaded log file!");
});

stream.on("data", chunk => {
    text += chunk;
});

stream.on("end", () => {
    let read = Buffer.byteLength(text, "utf-8");
    lastBytes += read;

    //Code here

});

Only reading additional content in a file

1 Answers1