gulp var defined in one pipe is undefined in another

Question

I am using gulp to processes pages for a site, some of these pages are php files. The problem is that after all the pages are run through the template engine they have a .html file extension. I am adding a property to the file that designates if it's supposed to be a file besides html and then renaming the file to match. However, for some reason gulp-rename keeps saying that the variable I am using to store the extension is undefined.

Corresponding task in gulpfile.js:

var gulp         = require('gulp'),
    gutil        = require('gulp-util'),
    lodash       = require('lodash'),
    data         = require('gulp-data'),
    filesize     = require('gulp-filesize'),
    frontMatter  = require('gulp-front-matter'),
    rename       = require('gulp-rename'),
    util         = require('util');    


gulp.task('metalsmith', function(){
var ext;                     //Variable to store file extension
return gulp.src(CONTENT_DIR)
    .pipe(frontMatter()).on("data", function(file){
        lodash.assign(file, file.frontMatter);
        delete file.frontMatter;
    })
    .pipe(data(function(file){                  //ext is defined here
        if(typeof file.filetype === 'undefined'){
            ext = {extname: '.html'};
        }else{
            ext = {extname: file.filetype};
        }
    }))
    .pipe(tap(function(file){
        console.log(ext);       //when I print to the console, ext is defined and correct
    }))
    .pipe(rename(ext)) //ext is undefined
    .pipe(gulp.dest(BUILD_DIR))
});

when I run the above it errors with Unsupported renaming parameter type supplied.

Also I have tried it having options obj on the same line as rename(), with ext only storing the file extension eg: .pipe(rename({extname: ext})) which causes the files to have undefined added as the extension (instead of phpfile.php the below md file would be named phpfileundefined)

yaml in phpfile.md

---
layout: php.hbt
filetype: ".php"
title: "php"
---

package.json

"devDependencies": {
    "gulp": "^3.9.1",
    "gulp-data": "^1.2.1",
    "gulp-filter": "^4.0.0",
    "gulp-front-matter": "^1.3.0",
    "gulp-rename": "^1.2.2",
    "gulp-util": "^3.0.7",
    "lodash": "^4.11.1"
  }

1) Your title says `ext` is defined at some point and undefined in another. What points are these? 2) There's a ton going on in this task, and without reading through each plugin's documentation we can't know whether there are any contributing factor syntax errors etc. Is it possible to pare this task down to something more minimal that still gives you the error, or are they all essential for generating the error? — henry, Sep 08 '16 at 01:47
I edited my question to include comments at the the relevant points, I also took out the unnecessary code and clarified the exact problem I'm having — binary-idiot, Sep 08 '16 at 05:11
Awesome. Can you add the `requires` section of your gulpfile? — henry, Sep 08 '16 at 05:23
Done, I added all of the requires and dependencies that are referred to in this section of my gulpfile — binary-idiot, Sep 08 '16 at 05:38
I think my last question: do the the source php files have the extension `.php`? — henry, Sep 08 '16 at 06:06
The php files are created using my template engine (the md file is just yaml that tells the engine to put the php template there) The file contents are correct, but just like the other files it has `undefined` instead of the correct extension. — binary-idiot, Sep 08 '16 at 06:12
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/122880/discussion-between-henry-and-binary-idiot). — henry, Sep 08 '16 at 06:13

score 2 · Accepted Answer · edited May 23 '17 at 11:51

Your problem is that by the time you execute rename(ext) the value of ext is still undefined because your data(...) code hasn't run yet.

Gulp plugins work like this:

You invoke the plugin function and pass it any necessary parameter. In your case rename(ext).
The plugin function returns a duplex stream
You pass that duplex stream to .pipe()
Only after all .pipe() calls have been executed does you stream start running.

So rename(ext) is a function call that is used to construct the stream itself. Only after the stream has been constructed can the stream start running.

However you set value of ext only once the stream is running. You need the value of ext before that when you are constructing the stream.

The easiest solution in your case is to simply do the renaming manually in your data(...) function instead of relying on gulp-rename. You can use the node.js built-in path module for that (which is what gulp-rename uses under the hood):

var path = require('path');

gulp.task('metalsmith', function(){
  return gulp.src(CONTENT_DIR)
    .pipe(frontMatter()).on("data", function(file){
        lodash.assign(file, file.frontMatter);
        delete file.frontMatter;
    })
    .pipe(data(function(file){
        var ext;
        if(typeof file.filetype === 'undefined'){
            ext = '.html';
        }else{
            ext = file.filetype;
        }
        var parsedPath = path.parse(file.path);
        file.path = path.join(parsedPath.dir, parsedPath.name + ext);
    }))
    .pipe(gulp.dest(BUILD_DIR))
});

Alternatively you could also use gulp-foreach as I suggest in this answer to a similar question. However that's not really necessary in your case since you're already accessing the file object directly in data(...).

EDIT: for completeness sake here's a version using gulp-foreach:

var foreach = require('gulp-foreach');

gulp.task('metalsmith', function(){
   return gulp.src(CONTENT_DIR)
    .pipe(frontMatter()).on("data", function(file){
        lodash.assign(file, file.frontMatter);
        delete file.frontMatter;
    })
    .pipe(foreach(function(stream, file){
        var ext;
        if(typeof file.filetype === 'undefined'){
            ext = {extname: '.html'};
        }else{
            ext = {extname: file.filetype};
        }
        return stream.pipe(rename(ext));
    }))
    .pipe(gulp.dest(BUILD_DIR))
});

I was under the impression that when you return the gulp task eg: `return gulp.src()...` it will run in sequence and that it will only run as you said when you dont return it — binary-idiot, Sep 08 '16 at 06:38
That's for the order of execution of **tasks**. That has nothing to do with the way streams are constructed or processed. — Sven Schoenung, Sep 08 '16 at 06:40

score 1 · Answer 2 · answered Sep 08 '16 at 07:18

@Sven's answer looks like the way to go, since you're hoping for something generalizable to other sites and since you can't drop the gulp-front-matter pipe.

Just to finish going over the code you had: Even

var ext = {extname: '.html'};
…
.pipe(rename(ext))
…

wouldn't work: gulp-rename would say "I don't know how to deal with the kind of data I was passed." You'd need to do

var ext = '.html';
…
.pipe(rename({extname: ext})
…

(or ext = 'html' and {extname: '.' + ext}, for example if the "html" string was being pulled from somewhere else and didn't have the ".")

As we discussed in chat, I was hoping to find that your php and html could be easily distinguished by file name or path. Since all your php files are in /blog you could use gulp-filter. Probably in the end it's not the best solution for this scenario, but it's a useful tool to know about so here's what it would look like:

var gulp = require('gulp'),
    gutil = require('gulp-util'),
    lodash = require('lodash'),
    data = require('gulp-data'),
    filesize = require('gulp-filesize'),
    filter = require('gulp-filter'),           // ADDED
    frontMatter = require('gulp-front-matter'),
    rename = require('gulp-rename'),
    util = require('util');

gulp.task('metalsmith', function() {
    const filterPHP = filter('blog/**/*', { restore: true });
    const filterHTML = filter('!blog/**/*', { restore: true });
    return gulp.src(CONTENT_DIR)

                                                  //---- if the only purpose of this particular gulp-front-matter pipe was to support the extension assignment, you could drop it
        .pipe(frontMatter()).on("data", function(file) {    //
            lodash.assign(file, file.frontMatter);          //
            delete file.frontMatter;                        //
        })                                                  //
                                                 //---------

        .pipe(filterPHP)                    // narrow down to just the files matched by filterPHP
        .pipe(rename({ extname: '.php' }))
        .pipe(filterPHP.restore)           // widen back up to the full gulp.src
        .pipe(filterHTML)                  // narrow down to just the files matched by filterHTML
        .pipe(rename({ extname: '.html' }))
        .pipe(filterHTML.restore)          // widen back up
        .pipe(gulp.dest(BUILD_DIR))
});

gulp var defined in one pipe is undefined in another

2 Answers2