How to pass an argument which exceeds length of 4000 from Javascript to Python using python shell

Question

I'm building a desktop application using node.js,electron.js,Vue.js and Python. I'm im using python to do some printing. I'm using the python-shell npm library (https://www.npmjs.com/package/python-shell). From python I'm using the pywin32 and sys libraries to print and to get the arguments i pass from JavaScript code. In my JavaScript code I have done as below and my output has more than 4000 characters. This the string i want to get printed. but not all the lines are getting printed. Is there a character limit when passing the values? If so how can i know the limit and can someone suggest me a better path to do this

const PythonShell = require("python-shell");

var options = {          
        pythonOptions: ['-u'],
        scriptPath: __dirname,
        args: [printerName, output, imgPath]
};

PythonShell.run("../../../../extras/script/printer.py", options, function (err, results) {
    if (err) throw err;
    for(var k in results){
        console.log(results[k])
    }              
});

Answer 1

The command line has a maximum size, which depends on your OS (and possibly your settings).¹ That command line has to be enough to fit the name of the Python interpreter, and your script path, and the -u, and all of your (properly quoted) args. It's probably 4096 on your system, so when you have around 4000 characters worth of arguments, things start failing.

There are two general solutions to this.

---

First, if the script uses arguments by looping over them and handing them independently, as many Unix tools do, you can call it repeatedly with, say, 50 arguments at a time. This is basically the same trick xargs uses for huge argument lists.

---

Second, you can change your script to read its input from a file, or from standard input, instead of from command-line arguments.

You haven't shown us your code, but let's say it looked like this:

import sys
for arg in sys.argv[1:]:
    print('arg:', arg)

You could change it to this:

import sys
for line in sys.stdin:
    print('arg:', line, end='')

But now you have to change the calling side. I don't know PythonShell, and my JS is pretty rusty, but from a quick look at the docs examples, it seems like it should look something like this:

var options = {          
        pythonOptions: ['-u'],
        scriptPath: 'myscript.py',
};

myscript = new PythonShell(options);
var results = [];
myscript.on('message', function(message)) {
    results += [message];
}
for (var i in args) {
    myscript.send(args[i] + '\n');
}
myscript.end(function(err, code, signal) {    
    if (err) throw err;
    for (var k in results){
        console.log(results[k])
    }              
});

---

Using a file instead of stdin is pretty similar. It would look something like this:

import sys
filename = sys.argv[1]
with open(filename) as f:
    for line in f:
        print('arg:', line, end='')

And then calling it is closer to your original code:

tempFile = new IDontRememberHowToSecurelyMakeATempFileInNode();
for (var i in args) {
    tempFile.write(args[i] + '\n');
}

var options = {          
    pythonOptions: ['-u'],
    scriptPath: __dirname,
    args: [tempFile.howeverYouGetTheFileName()]
};

PythonShell.run("../../../../extras/script/printer.py", options, function (err, results) {
    if (err) throw err;
    for(var k in results){
        console.log(results[k])
    }              
});

---

_{1. On Windows, it's more complicated than this. In some versions, the command-line arguments and set of environment variables together have to be under another limit, and the limits may be different for UTF-16 vs. 8-bit inputs, and there are additional limits if you're using the MSVCRT functions that emulate POSIX-style APIs instead of calling the Windows APIs directly, etc.}

Answer 2

As you can see in python-shell documentation you can use pyshell.send() to send data to python stdin - that would not have the limitation.