0

I am parsing a JSON file using a parsing stream module and would like to stream results to request like this:

var request = require("request")

fs.createReadStream('./data.json')
  .pipe(parser)
  .pipe(streamer)
  .pipe(new KeyFilter({find:"URL"}) )
  .pipe(request)
  .pipe( etc ... )

(Note: KeyFilter is a custom transform that works fine when piping to process.stdout)

I've read the docs and source code. This won't work with 'request' or 'new request()' because the constructor wants a URL.

jdungan
  • 243
  • 2
  • 10
  • Stream results to *where*? Request module makes HTTP calls, which is why it wants a URL. – laggingreflex Jun 11 '15 at 20:06
  • I'm trying to batch process a list of urls from the JSON file to request and then on to additional parsing streams. I could just emit a 'found a url' event but that seems like an inelegant break in the process. – jdungan Jun 12 '15 at 00:40

2 Answers2

0

It will work with request.put() as this : yourStream.pipe(request.put('http://example.com/form'))

Pierre Inglebert
  • 3,858
  • 1
  • 18
  • 22
  • Thanks, but this only calls 'example.com' once and then fails. The goal is to have a stream of urls piping into request with its results piping onward for additional parsing. – jdungan Jun 12 '15 at 00:32
0

After more research and experimenting I've concluded that request cannot be used in this way. The simple answer is that request creates a readable stream and .pipe() methods expects a writable stream.

I tried several attempts to wrap request in a transform to get by this with no luck. While you can receive the piped url and create a new request, I can't figure out how to reset the pipe callbacks without some truly unnatural bending of the stream pattern. Any thoughts would be appreciated, but I have moved on to using an event in the url stream to kick off a new request(url).pipe(etc) type stream.

jdungan
  • 243
  • 2
  • 10