Stream binary file with http post

Question

I'm using the request library to send a binary (pdf) file in the body of the request using http post (NOTE: This API does not accept multi-part forms). However, I have only been able to get it to work using fs.readFilesync(). For some reason, when I try to use fs.createReadStream() the pdf file is still sent, but it is empty, and the request never finishes (I never get a response back from the server).

Here is my working version using fs.readFileSync():

const request = require('request');
const fs = require('fs');

const filename = 'test.pdf';

request({
    url: 'http://localhost:8083/api/v1/endpoint',
    method: 'POST',
    headers: {
        'Content-Type': 'application/octet-stream',
        'Accept': 'application/vnd.api+json',
        'Content-Disposition': `file; filename="${filename}"`
    },
    encoding: null,
    body: fs.readFileSync(filename)
}, (error, response, body) => {
    if (error) {
        console.log('error:', error);
    } else {
        console.log(JSON.parse(response.body.toString()));
    }
});

If I try to replace the body with the below, it doesn't work:

body: fs.createReadStream(filename)

I have also tried piping the http request on to the stream, like it says in the request library docs, but I get the same result:

fs.createReadStream(filename).pipe(request({...}))

I've tried to monitor the stream by doing the following:

var upload = fs.createReadStream('test.pdf');

upload.pipe(req);

var upload_progress = 0;
upload.on("data", function (chunk) {
    upload_progress += chunk.length
    console.log(new Date(), upload_progress);
})

upload.on("end", function (res) {
    console.log('Finished');
    req.end();
})

I see progress for the stream and Finished, but still no response is returned from the API.

I'd prefer to create a read stream because of the benefits of working better with larger files, but am clueless as to what is going wrong. I am making sure I'm not altering the file with any special encoding as well.

Is there some way to get some kind of output to see what process is taking forever?

UPDATE:

I decided to test with a simple 1 KB .txt file. I found that it is still empty using fs.createReadStream(), however, this time I got a response back from the server. The test PDF I'm working with is 363 KB, which isn't outrageous in size, but still... weren't streams made for large files anyway? Using fs.readFileSync() also worked fine for the text file.

I'm beginning to wonder if this is an synchronous vs asynchronous issue. I know that fs.readFileSync() is synchronous. Do I need to wait until fs.createReadStream() finishes before trying to append it to the body?

score 3 · Accepted Answer · answered Dec 04 '19 at 17:06

I was able to get this working by doing the following:

const request = require('request');
const fs = require('fs');

const filename = 'test.pdf';

const readStream = fs.createReadStream(filename);

let chunks = [];
readStream.on('data', (chunk) => chunks.push(chunk));
readStream.on('end', () => {
    const data = Buffer.concat(chunks);

    request({
        url: 'http://localhost:8083/api/v1/endpoint',
        method: 'POST',
        headers: {
            'Content-Type': 'application/octet-stream',
            'Accept': 'application/vnd.api+json',
            'Content-Disposition': `file; filename="${filename}"`
        },
        encoding: null,
        body: data
    }, (error, response, body) => {
        if (error) {
            console.log('error:', error);
        } else {
            console.log(JSON.parse(response.body.toString()));
        }
    });
});

I chunked the data together and concatenated it with a buffer before making the request.

I noticed in the documentation it said this:

The Buffer class was introduced as part of the Node.js API to enable interaction with octet streams in TCP streams, file system operations, and other contexts.

The API I'm calling requires the application/octet-stream header, so I need to use the buffer rather than streaming it directly.

Stream binary file with http post

1 Answers1