0

I have an array of parameters, which I then transform into an array of urls. I would like to then stream this array of urls to request.

var parameters = [1,2,.., n]
var urls = [];
parameters.forEach(function(item) {
    urls.push('http://www.example.com/page'+item)
})

Using the module event-stream I can transform this array into a stream like this:

var es = require('event-stream');
var reader = es.readArray(urls);

I can then do: 

reader.pipe(process.stdout)

I know how to use request so I can pipe its result like this:

 request(url).pipe(csvToJson).pipe(fs.createWriteStream('destinationFile'))

But how can I stream the array of urls into request, if that's possible?

Bondifrench
  • 1,272
  • 1
  • 20
  • 35
  • Make `csvToJson` some `data` object and then maybe key them like `var data = { csv: csvToJson, urls: urls }` – Kevin Reilly Sep 18 '14 at 10:46
  • I may have been unclear, basically given an array of urls I would like to `urls.forEach(function(url) { request(url)}).pipe(csvToJson).pipe(transform).pipe(destination)` My problem is how to pipe the urls individually into the request function/module? – Bondifrench Sep 20 '14 at 06:20

0 Answers0