Problems in writing binary data with node.js

Question

I am trying to write the binary body of a request to a file and failing. The file is created on server but I am unable to open it. I am getting 'Fatal error: Not a png' on Ubuntu. Here is how I am making the request:

curl --request POST --data-binary "@abc.png" 192.168.1.38:8080

And here is how I am trying to save it with the file. The first snippet is a middleware for appending all the data together and second one is the request handler:

Middleware:

app.use(function(req, res, next) {
  req.rawBody = '';
  req.setEncoding('utf-8');
  req.on('data', function(chunk) { 
    req.rawBody += chunk;
  });
  req.on('end', function() {
    next();
  });
});

Handler:

exports.save_image = function (req, res) {
  fs.writeFile("./1.png", req.rawBody, function(err) {
    if(err) {
      console.log(err);
    } else {
      console.log("The file was saved!");
    }
  });
  res.writeHead(200);
  res.end('OK\n');
};

Here's some info which might help. In the middleware, if I log the length of rawBody, it looks to be correct. I am really stuck at how to correctly save the file. All I need is a nudge in the right direction.

Can you clarify "failing"? Does the file exist on the server but have incorrect content? Does the file not exist at all on the server? Do you get an error message? — Peter Lyons, Mar 19 '14 at 13:14

score 4 · Accepted Answer · answered Mar 19 '14 at 13:38

4

Here is a complete express app that works. I hit it with curl --data-binary @photo.jpg localhost:9200 and it works fine.

var app = require("express")();
var fs = require("fs");
app.post("/", function (req, res) {
  var outStream = fs.createWriteStream("/tmp/upload.jpg");
  req.pipe(outStream);
  res.send();
});
app.listen(9200);

I would just pipe the request straight to the filesystem. As to your actual problem, my first guess is req.setEncoding('utf-8'); as utf-8 is for text data not binary data.

answered Mar 19 '14 at 13:38

Peter Lyons

142,938
30
279
274

Yes it is working. However, in some cases, I am only getting partial image. This generally happens over a remote connection. – Akash Mar 19 '14 at 13:50
Well, that's probably not a problem you can fix in your code. If the network doesn't deliver the packets, nothing you can do in your application code is going to magically synthesize them. You'd need to dig in there to discover exactly what is going wrong, but it's probably not a code problem. – Peter Lyons Mar 19 '14 at 14:30
Maybe you should check content-length header against the incoming data, if not matching reject. – Farid Nouri Neshat Mar 19 '14 at 14:32
@FaridNouriNeshat If I reject a chunk, I don't think I will be able to assemble full image. I am looking at node stream. Peter's answer will probably nail it for me. – Akash Mar 19 '14 at 15:43
I meant reject the whole request, if see the end event emitted but received content is not complete, so you just assume the request was ended half finished and don't process it further, server the user with a 4xx status code. – Farid Nouri Neshat Mar 19 '14 at 23:34
Maybe the problem is that you're calling `res.send();` too soon. It could be that the client receives the response before it has finished sending the request and stops sending. – David Knipe Mar 31 '18 at 09:33
...or, perhaps more accurately, the server closes the connection before the client has finished sending. – David Knipe Mar 31 '18 at 16:24
And if that is the problem, then I think you can fix it by replacing `res.send();` with `outStream.on('close', function() { res.send(); });`. – David Knipe Mar 31 '18 at 20:04

score 0 · Answer 2 · answered Mar 19 '14 at 14:22

For fix your code: I'm with @Peter Lyons, that the error is probably the req.setEncoding('utf-8'); line.

I know the following don't ask your question directly, but proposes an alternative to it by using req.files functionality provided by Express.js, which you are using.

if (req.files.photo && req.files.photo.name) { // Get the temporary location of the file. var tmp_path = req.files.photo.path;

// set where the file should actually exists - in this case it is in the "images" directory.
var target_path = './public/profile/' + req.files.photo.name;

// Move the file from the temporary location to the intended location.
fs.rename(tmp_path, target_path, function (error) {
    if (!error) {
        /*
         * Remove old photo from fs.
         * You can remove the following if you want to.
         */
        fs.unlink('./public/profile/' + old_photo, function () {
            if (error) {
                callback_parallel(error);
            }
            else {
                callback_parallel(null);
            }
        });
    }
    else {
        callback_parallel(error);
    }
});

}

This would have been my preference too but I have to work with an external source of data provided to me which is beyond my control. Thanks for answer. — Akash, Mar 19 '14 at 15:37
req.files property will be defined if the client used an http form with enctype="multipart/form_data" and then issued a submit on that form. If the client used a different mechanism, for instance, an AJAX call, there will be no such property set. — Alex, Mar 28 '14 at 22:01

Problems in writing binary data with node.js

2 Answers2