Using asynchronous Node.js functions with Python subprocesses

Question

I have an app.js file that uses fs.watchFile() to check for changes in a text file. The changes are then outputed to the console. I have a python.py that I would like to use this JS file while the same time running code for the GPIO pins on a RasberryPi (the GPIO code is not shown here for simplicity). When running python.py the program gets stuck after priting "after start" which means that the JS file is at fault. I have tried using Python asyncio as well but I've received the same result as when I don't use it. Is there a way of allowing an asynchronous process on Node.js to run, while letting be a Python subprocess and allow the rest of the code to continue normally? Should I try running Python using Node.js instead?

app.js:

const fs = require('fs');

function main() {
    const text_file= '/home/user/git/corp/file.txt';
    const text = fs.readFileSync(policies_file, "utf8");
    console.log(text);    
    fs.watchFile(text_file, {bigint: false, persistent: true, interval: 1}, (curr, prev) => { 
            const text= fs.readFileSync(text_file, "utf8");
            console.log(text);
    }
main();

python.py

import subprocess
import asyncio

async def subproc():
    print("start subprocess")
    proc = subprocess.Popen(["/usr/bin/node","/home/user/git/corp/prudens-js/node/app.js", ], stdout=subprocess.PIPE)
    print("after start")
    out = proc.communicate()
    print("after finish")
    print(out)  

async def main():
    print("start")
    await asyncio.gather(subproc())
    print("waiting")

asyncio.run(main())

Kent Gibson · Answer 1 · 2023-07-13T13:05:31.997

1

"the program gets stuck after priting "after start" which means that the JS file is at fault."

Not at all. The problem is that proc.communicate() is blocking, even inside an async function, so it will block python.py until app.js completes - which it never does.

If you want to read output from the process as it is produced use proc.stdout.readline() instead.

Don't use async for this, use threads - one for the subprocess communication and another for the GPIO.

e.g.

import subprocess
import threading

def subproc():
    print("start subprocess")
    proc = subprocess.Popen(["/usr/bin/node", "app.js", ], stdout=subprocess.PIPE)
    print("after start")
    while 1:
        out = proc.stdout.readline()
        print("out:", out)
        
def main():
    t = threading.Thread(target=subproc)
    t.start()
    print("thread started")
    # do other stuff here in the main thread
    
    # but I'll just wait on the thread - which will never complete
    t.join()
    print("thread complete")

main()

edited Jul 13 '23 at 13:05

answered Jul 12 '23 at 12:37

Kent Gibson

101
1
5

`app.js` should output something since I'm using `fs.watch()` and changing the files and but is happening. It's definitely being blocked as you said. Any tips as to how to use threads? Thank you for your reply! – Yiannis Michael Jul 12 '23 at 14:33
It was even worse than I thought, as your app.js never completes, so `communicate` would block indefinitely. Anyway, updated the answer with `readline` and threads - not sure if that is what you are after, but it works for me. – Kent Gibson Jul 13 '23 at 13:07
When I try to implement it I will mark it as solved or ask you to clarify further! Thank you! – Yiannis Michael Jul 17 '23 at 07:53

Using asynchronous Node.js functions with Python subprocesses

1 Answers1