0

I'm thinking whether it is possible to pass an array as input of awk command via Node.JS child_process.spawn().
I'll explain myself with an example.

Let's define an array (in my real case the array has about 3 million and something rows) : 
const myvar = ['val1', 'val2', 'val1', 'val3', 'another', 'etc', 'etc'];

Now, using child_process, I want to pass this variable as input of awk command, remove duplicates and save the output into a text file.
Wrongly, I'm currently using a template string, thinking that maybe the command would read from the variable:

const { spawn } = require('child_process');

const awk = spawn(
    `awk '!a[$0]++' ${myvar} > outputfile.txt`
);

but it's obviously a wrong usage. Since, I get:

  • awk exits with code 2 (wrong usage)

or also:

  • I get Error: spawn E2BIG
I'm finding myself stuck in this issue. I know I can pass a simple string variable but it's different.

.

Is something like this doable?

N.B.: the array is taken from a text file loaded into the code at an earlier stage. That's why I don't want to read from the file again. Using the text file as input works okay with awk but I'm trying to avoid reading that multiple times.

If I didn't explain clearly, please let me know. Thanks for help!

Marco
  • 77
  • 1
  • 10

1 Answers1

1
['val1', 'val2', 'val1', 'val3', 'another', 'etc', 'etc']

————————————————————————————————————

{m/g}awk 
'ORS = __[$-_]++ < /[^\47]/ ? "\n" :_'  \
  FS = '^$'                              \
  RS = '[[][\47]|[\47]([,][ \t]*[\47]|[]][\n]?)'

or these really fringe syntax style :

'ORS = __[$-_]-->-/[^\47]/ ? "\n" : _'

'ORS = -/[^\47]/~--__[$-_] ? "\n" : _'

————————————————————————————————————

val1
val2
val3
another
etc
RARE Kpop Manifesto
  • 2,453
  • 3
  • 11
  • Thanks for your reply. So you're using built-in variables (ORS, FS, RS) within the awk command, sure. But I cannot see how I'm supposed to use that. How should I input my array variable and output a text file, in this case? And why the ternary operator for the input/output? – Marco May 23 '22 at 14:59
  • if you look at tthat code carefully, i'm not "using" FS and RS inside the command - i'm just setting them after the command syntax (but still runs before the command itself). just dump out your array exactly looking like : "['val1', 'val2', 'val1', 'val3', 'another', 'etc', 'etc']", like in your example, and the code will handle the transformation and de-dupe – RARE Kpop Manifesto May 24 '22 at 02:43
  • or you can show me what the default look is you just directly dump out an array to console, and i can adjust my code for that. i never worked with NodeJS before so you'll need to help me out – RARE Kpop Manifesto May 24 '22 at 02:44
  • ok, so I defined a variable `myvar` which includes the array that we know. Inside my **Node.JS** script, I use a library which allows me to call the `awk` command as a string (well, not only but in this case is what I need). Basically I should include in this awk command the variable as input and in the same time asking `awk` to output a text file. In general I would write as `awk '{do something} variable_input > text_output.txt` but now I cannot find a way to achieve this. If you're not used to Node.JS, how can I parse a variable AND in the same time output a text file in plain awk? – Marco May 24 '22 at 08:18
  • Okay, I "translated" your awk command and I think now it's clear. The problem that persists is the input and output. – Marco May 24 '22 at 08:56
  • maybe try : const awk = spawn( ` printf '%s' "${myvar}" | mawk 'awk_code_here' > outputfile.txt` ); – RARE Kpop Manifesto May 24 '22 at 10:31
  • you're just trying to de-dupe the data values right ? why do you need awk for that ? Doesn't NodeJS have something appropriate for something like that ? – RARE Kpop Manifesto May 24 '22 at 10:35
  • Yes it has but I need the script to run as fast as possible and I tested the pure NodeJS way and takes way more time. Thanks for your try, I'll see if it's gonna work if yes I'll accept your answer :) – Marco May 24 '22 at 10:40
  • but it *has* to be javascript ? yikes – RARE Kpop Manifesto May 24 '22 at 10:49