Node and Lazy: How do I know when it's done?

Question

I need to read a file line by line, and change a variable accordingly. I would normally write this in PHP... but I decided to take the challenge.

I wrote:

fs = require('fs');
Lazy = require('lazy');
path = require('path');

files = fs.readdirSync('.');
var software = {};


files.forEach( function(fileName){


  var m; 
  if( m = fileName.match(/^(.*)\.txt$/) ){
    name = m[1];

    console.log("Processing file: " + fileName);
    software[name] = {};
    console.log("Software 1: %j",software);

    var section = 'unset';
    new Lazy(fs.createReadStream(fileName)).lines.forEach(
      function(line){
        var m;
        line = line + '';
        if( m = line.match(/^([a-zA-Z_]*):$/)){
          section = m[1];
          software[name][section] = '';
          console.log("Switching to section " + m[1]);
          console.log("Software 2: %j",software);
        } else if (line == '.'){
          section = 'unset'
        } else if (line == ''){
          section = 'unset'
        } else { 
          console.log("LINE: " + line) ;
          software[name][section] = software[name][section] + line + "\n";
          console.log("Software 3: %j",software);
        }
      }

    );   
  }

});

console.log("Software 4: %j",software);

Apart from the code being very ugly and very unoptimised, I am having trouble as when the last line prints, the "software" variable is not YET populated! I am guessing Lazy is asyncronous. So, it basically works, but "at some point later". This is great, but... where do I write code when that important cycle, that fills in the software variable, is actually finished?!?

As requested: data to play with!

simply create "something.txt" and write:

name:
Name 1
.

Option 1:
Value 1
.

Option 2:
Value 2
.

Option 3:
Multi
Line
Value
.

Another_section:
Again
.

Merc.

Good question! Could you give me some short example data to play with? — jsalonen, Jun 20 '12 at 11:09
I mean, I *could* store fs.createReadStream(fileName)).lines.length and increase a counter for each line and then when the number of line is reached call a function... but it seems completely crazy to do. SURELY! — Merc, Jun 20 '12 at 11:15

Michelle Tilley · Accepted Answer · 2012-06-20T11:40:18.193

15

The instances of Lazy returned by the library are EventEmitters, and it emits en event called pipe when a "set" of operations is complete:

new Lazy(
  ...
).on('pipe', function() {
  // all done
});

Modifying your code to use this event results in (the only change is near the bottom):

fs = require('fs');
Lazy = require('lazy');
path = require('path');

files = fs.readdirSync('.');
var software = {};


files.forEach( function(fileName){


  var m;
  if( m = fileName.match(/^(.*)\.txt$/) ){
    name = m[1];

    console.log("Processing file: " + fileName);
    software[name] = {};
    console.log("Software 1: %j",software);

    var section = 'unset';
    new Lazy(fs.createReadStream(fileName)).lines.forEach(
      function(line){
        var m;
        line = line + '';
        if( m = line.match(/^([a-zA-Z_]*):$/)){
          section = m[1];
          software[name][section] = '';
          console.log("Switching to section " + m[1]);
          console.log("Software 2: %j",software);
        } else if (line == '.'){
          section = 'unset'
        } else if (line == ''){
          section = 'unset'
        } else {
          console.log("LINE: " + line) ;
          software[name][section] = software[name][section] + line + "\n";
          console.log("Software 3: %j",software);
        }
      }

    ).on('pipe', function() {
      console.log("Software 4: %j",software);
    });
  }

});

[Edit] To answer your question regarding how I found this info:

I did indeed check out the source file for the project; I knew the library had a sum method that could be chained to instances of Lazy to sum up everything at the end; the code for that method calls foldr, and the code for that method listens for an event called pipeName, which is defaulted in line 22 as pipe.

edited Jun 20 '12 at 11:40

answered Jun 20 '12 at 11:30

Michelle Tilley

157,729
40
374
311

I _have_ to ask... how did you find out?!? I read Lasy's docs, and didn't figure it out from there. THere isn't much in there in terms of "pipe" or "event" or "emit" either... I mean, how did you find this out? Did you have to dive into the source code? – Merc Jun 20 '12 at 11:34
1

Yeah, I agree the docs aren't very comprehensive; I greatly prefer `async` as my flow control library of choice. I've updated my answer with some additional info. – Michelle Tilley Jun 20 '12 at 11:40
OK but my question stands... how did you find out?!? :D – Merc Jun 20 '12 at 11:42
@Merc: He added explanation to the end of his answer. – jsalonen Jun 20 '12 at 11:43
You mean how did I know it would be in there? Well, a flow control library that doesn't tell you when it's done isn't very useful, so I kind of made an assumption that something would be in there--besides, the chainable methods like `sum`, etc. wouldn't work in that case. It was just luck that the implementation was an event that [gets re-emitted](https://github.com/pkrumins/node-lazy/blob/83afb5e6ab991237fe1a0ee4f0b2ac2961eb5cdf/lazy.js#L29). – Michelle Tilley Jun 20 '12 at 11:44
I'm getting instances of `pipe` firing before `foreach` is complete. I'm using `foreach` to read a file in, and modify specific lines, and then `pipe` to write back to the file. In some circumstances this results in either desired behaviour, a half empty file, or a totally empty file. – DanH Feb 15 '14 at 02:05
I tried https://github.com/nickewing/line-reader instead which seems much more purpose built and does not exhibit the race condition observed. – DanH Feb 15 '14 at 02:18

Node and Lazy: How do I know when it's done?

1 Answers1