To merge file in node js?

Question

I need to remove all lines except for the last 500 lines. And I will use the fs module. if you know how to do that, please tell me how to do that!

p.s I can't speak English well. I hope you understand my eassay.

Answer 1

I can only tell how to create a copy with the last N lines.

1. If your system has tail and you do not mind using child_process module:

import child_process from 'child_process';

child_process.exec('tail -n 500 test.txt > test2.txt');

or:

import child_process from 'child_process';
import fs from 'fs';

child_process.spawn(
  'tail', ['-n', '500', 'test.txt']
).stdout.pipe(fs.createWriteStream('test2.txt'));

2. If your file is not huge and you can read it as a whole into memory and use some intermediate structures:

import fs from 'fs';
const text = fs.readFileSync('test.txt', 'utf8');
const lines = text.split('\n');
fs.writeFileSync('test2.txt', lines.slice(-500).join('\n'));

3. If your file is huge, you can use readline module to read it line by line twice: first to count lines, second to write last N ones.

import fs from 'fs';
import readline from 'readline';

(async function main() {
  const counterStream = fs.createReadStream('test.txt');

  const counter = readline.createInterface({
    input: counterStream,
    crlfDelay: Infinity,
  });

  let lineCount = 0;
  for await (const line of counter) {
    lineCount++;
  }

  const startAfter = Math.max(0, lineCount - 500);

  const readerStream = fs.createReadStream('test.txt');
  const result = fs.openSync('test2.txt', 'a');

  const reader = readline.createInterface({
    input: readerStream,
    crlfDelay: Infinity,
  });

  lineCount = 0;
  for await (const line of reader) {
    if (++lineCount > startAfter) fs.writeSync(result, `${line}\n`);
  }
})();

Also see anwers to these questions:

Remove last n lines from file using nodejs

NodeJS - How to remove the first line of a text file without read all lines?

UPD: A (probably naive) attempt to emulate tail -n:

import fs from 'fs';

const path = 'test.txt';
const lastLinesLimit = 5;

const fileSize = fs.statSync(path).size;
const fd = fs.openSync(path);

// Increase the next number appropriately to decrease read calls.
let bufferSize = Math.min(10, fileSize);
let buffer = Buffer.alloc(bufferSize);

let stringTail = '';
let position = fileSize;

while (position !== 0) {
  position -= bufferSize;
  if (position < 0) {
    bufferSize += position;
    buffer = buffer.subarray(0, position);
    position = 0;
  }
  fs.readSync(fd, buffer, 0, bufferSize, position);
  stringTail = buffer.toString() + stringTail;
  if (stringTail.match(/\n(?!$)/g)?.length >= lastLinesLimit) break;
}

console.log(
  stringTail
    .split(/\n(?!$)/)
    .slice(-lastLinesLimit)
    .join('\n')
);