0

I'm using Node.js to extract some large files (sometimes larger than 2 GB).

Not all the data in these files are needed hence why, is order to avoid downloading the whole file, i'm currently using sshpass command combined with cat and grep on whatever i need from the file but i'm facing issues when the output of the cat | grep command is empty.

To be consistent, i want to stop using sshpass and instead use ssh2-promise package to execute the cat grep command and write the output of the command to a file to be processed, transformed into a csv file and loaded in a certain table in my database (mariadb)

The issue i'm having is that whenever the output of the command is larger than 512 MB the data gets truncated and i lose a lot of records. This is how i'm using the package.

import SSH2Promise = require('ssh2-promise');
const id = randomBytes(16).toString('hex');
const ssh = new SSH2Promise({ uniqueId: id, ...sshConfig });
const command = 'cat fileName | grep "match"'
const commandResult = await ssh.exec(command);
await fs.promises.writeFile(filePath, commandResult);

How can I ensure that the output of the ssh2-promise package isn't truncated when the output exceeds 512 MB?

I've considered handling these files in chunks and using grep on whatever i'm searching for, but doing so i have to make multiple ssh connections and some records will be lost.

The only logical answer that my research lead to is to increase the highWaterMark but it seems that this is not a valid option to pass as the output is not changing.

so any help would be appreciated.

nadimgeagea
  • 11
  • 1
  • Use code fences for code, not inline code blocks. Also you need to provide more details, more code, and describe you problem a little better. If you fix this post, other community members may be able to better help you out. – Viraj Shah Mar 17 '23 at 20:57
  • Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. – Community Mar 17 '23 at 20:57
  • "*in order to avoid downloading the whole file, i'm currently using `sshpass` command combined with `cat` and `grep`*" - you really should consider using nodejs streams instead – Bergi Mar 21 '23 at 13:14
  • i'm actually using sshpass so that i don't download the whole file in order to save space – nadimgeagea Mar 28 '23 at 12:19

0 Answers0