Nodejs fs.createReadStream event error

Question

I am trying to read data from a file very quickly (data from an ADC). The conversion is started on opening the file, and finished at close. I need to wait opening the file again, and wait for the currently conversion to complete.

My problem is, that when i am opening the file very quickly nodejs wont caught the expected events. Any clue how to fix this?

 node.on('input', function(){

            readStream = fs.createReadStream(path.location,{encoding: 'utf8'});

            if (readyFlag == 1) {
                readStream.on('data',(data) => {
                    data = {payload: data/1000};
                    node.send(data);
                    console.log(`data: ${data}`);

                })
            }

            readStream.on('open', () => {
                console.log("file opened");
                readyFlag = 0;
            })   



            readStream.on('close', () => {
                console.log("file closed");
                readyFlag = 1;
            })

            readStream.on('error', (err) => {
                console.log(err);
            })




    })

I end up with the file being opened all the time.

What are you actually trying to do here? Are you trying to protect multiple reads on the file? It's slightly confusing, as for why the file is remaining open, could be the fact you are passing options and not setting `autoClose: true` - it should default to that according to [the docs](https://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options) but that may only happen if you pass no options at all. — James, Jun 26 '18 at 14:19
I can see it's a bit confusing. Sorry! In fact yes, I am trying to protect from multiple reads. I want to be sure the file is closed, before another read can take action. My code works, when I open and close the file with in 1 sec. But if I force it to read every 0.1 sec, it only gets the 'open' event and never the 'close' event. — Simon, Jun 26 '18 at 16:50

score 0 · Accepted Answer · answered Jun 26 '18 at 19:05

0

So looks to me like the problem here is the context of readStream - it's not local to the input callback meaning each message will overwrite the reference to the previous stream.

The fact this starts to fail when you increase the frequency of the messages seems logical to me because the stream hasn't had a chance to finish so readyFlag is never reset.

The fix for this should literally a one liner, make the stream a local var

const readStream = ...

answered Jun 26 '18 at 19:05

James

80,725
18
167
237

Its more than this because `readyFlag` will always be 0 when the `'data'` event could registered for `readStream`, unless it is initialized to 1. Additionally, shouldn't `readyFlag` be scoped the same as `readStream` since it is a flag for that stream? – peteb Jun 26 '18 at 19:09
Thanks. Makes sense! I will try that tomorrow. To your comment: I have initialized the flag to be equal to 1. But you are right about the scope. I will change that as well tomorrow. Thanks! – Simon Jun 26 '18 at 19:29
@peteb I made the assumption `readyFlag` is being initialised somewhere else otherwise it wouldn't work at all... I think `readyFlag` needs to be scoped globally as each message coming in needs to work off that state. – James Jun 26 '18 at 19:30

score 0 · Answer 2 · answered Jun 27 '18 at 06:32

I think I have solved it. The readyFlag has to be global to the input scope, otherwise it would not work. I moved the 'open' event inside the the if statement, which is checking for the readyFlag. That seemed to do the problem. Now the system dont hang, and the file is only being opened, when its done reading and have closed the previously open section.

Here is the code:

    var readyFlag = 1;
    node.on('input', function(msg){

            const readStream = fs.createReadStream(path.location,{encoding: 'utf8'});


            if (readyFlag == 1) {
                readStream.on('data',(data) => {
                    data = {payload: data/1000};
                    node.send(data);
                    console.log('data read');
                    readyFlag = 0;

                })

                readStream.on('open', () => {
                    console.log("file opened");

                })   
            }

            readStream.on('close', () => {
                console.log("file closed");
                readyFlag = 1;
            })

            readStream.on('error', (err) => {
                console.log(err);
            })
    })

Not sure I agree with this - are you saying if you don't make `readStream` a local var this works? I can't see why moving the `readStream.on('open', ...)` into the `if` on it's own will make any sort of difference here, the fix was per my answer to make `readStream` a local var. — James, Jun 27 '18 at 12:14
Hi @James I have been working on this the whole day. Unfortunately my solution did not work as expected. And of course it does not! I'm fairly new to JavaScript an node.js in general, so bare on me.. I have now understand, that the readyFlag dont have any role in this context, as all actions in the if statements is asynchronous. Well.. just removed the check for readyFlag. The main problem is, that node-red flow editor web service is going down, if I use the node.send(data) object. If I only output it on the console.log no problems at all — Simon, Jun 27 '18 at 19:03
sounds like Node-Red is crashing, hard to say what could be causing that as could be another node given it happens on `node.send` - however, if the file is no longer being left open sounds to me like the problem detailed in the question has been fixed as a result of the local var. — James, Jun 27 '18 at 19:40

Nodejs fs.createReadStream event error

2 Answers2