2

Evolving from Javascript, spliced FileReader for large files with Promises, how?, which showed me how a Promise can also resolve a function, now I am stuck with the same but inside an Array.reduce function.

The goal is that I want to upload a file (which already does) within an array, where each array item (a file) is uploaded sequentially (i.e. controlled through promises).

Then, I understand that the answer is somehow in http://www.html5rocks.com/en/tutorials/es6/promises/?redirect_from_locale=es , but I cannot understand how to apply that to here. My array is not an array of promises, is an array of files. Well, the whole thing is still obfuscated to me.

This is my code, which would work if I could see the ein console.log message:

return myArray.reduce(function(previous, current) {
    var BYTES_PER_CHUNK = 100000;
    var start = 0;
    var temp_end = start + BYTES_PER_CHUNK;
    var end = parseInt(current.size);
    if (temp_end > end) temp_end = end;
    var content = ''; // to be filled by the content of the file
    var uploading_file = current;
    Promise.resolve().then(function() {
        return upload();
    })
    .then(function(content){
        // do stuff with the content
        Promise.resolve();
    });
},0)  // I add the 0 in case myArray has only 1 item
//},Promise.resolve()) goes here?

.then(function(){
    console.log('ein') // this I never see
});

function upload() {
  if (start < end) {
    return new Promise(function(resolve){
      var chunk = uploading_file.slice(start, temp_end);
      var reader = new FileReader();
      reader.readAsArrayBuffer(chunk);
      reader.onload = function(e) {
        if (e.target.readyState == 2) {
          content += new TextDecoder("utf-8").decode(e.target.result);
          start = temp_end;
          temp_end = start + BYTES_PER_CHUNK;
          if (temp_end > end) temp_end = end;
          resolve(upload());
        }
      }
    });
  } else {
    uploading_file = null;
    return Promise.resolve(content);
  }
}
  • updated after several comments, it seems that now it works ... not sure yet

    var uploading_file, start, temp_end, end, content; var BYTES_PER_CHUNK = 100000;

    myArray.reduce(function(previous, current) { return previous .then(function() { BYTES_PER_CHUNK = 100000; start = 0; temp_end = start + BYTES_PER_CHUNK; end = parseInt(current.size); if (temp_end > end) temp_end = end; content = ''; uploading_file = current;

    upload()
    .then(function(content){
        // do stuff with "content"
        console.log('here')
        return Promise.resolve();
    });
    

    }); },Promise.resolve()) .then(function(){ console.log('ein'); });

    function upload() { if (start < end) { return new Promise(function(resolve){ var chunk = uploading_file.slice(start, temp_end); var reader = new FileReader(); reader.readAsArrayBuffer(chunk); reader.onload = function(e) { if (e.target.readyState == 2) { content += new TextDecoder("utf-8").decode(e.target.result); start = temp_end; temp_end = start + BYTES_PER_CHUNK; if (temp_end > end) temp_end = end; resolve(upload()); } } }); } else { uploading_file = null; return Promise.resolve(content); } }

  • improved code, seems to work, perhaps easier to read?

        var start, temp_end, end;
        var BYTES_PER_CHUNK = 100000;
    
        myArray.reduce(function(previous, current) {
            return previous
            .then(function() {
                start = 0;
                temp_end = start + BYTES_PER_CHUNK;
                end = parseInt(current.size);
                if (temp_end > end) temp_end = end;
                current.data = '';
    
                return upload(current)
                .then(function(){
                    // do other stuff
                    return Promise.resolve();
                });
            });
        },Promise.resolve())
        .then(function(){
          // do other stuff
        });
    
        function upload(current) {
            if (start < end) {
                return new Promise(function(resolve){
                    var chunk = current.slice(start, temp_end);
                    var reader = new FileReader();
                    reader.readAsText(chunk);
                    reader.onload = function(e) {
                        if (e.target.readyState == 2) {
                            current.data += e.target.result;
                            start = temp_end;
                            temp_end = start + BYTES_PER_CHUNK;
                            if (temp_end > end) temp_end = end;
                            resolve(upload(current));
                        }
                    }
                });
            } else {
                return Promise.resolve();
            }
        }
    
Community
  • 1
  • 1
GWorking
  • 4,011
  • 10
  • 49
  • 90
  • 1
    What is the point of using Array reduce if you never even refer to the `previous` parameter - you're basically doing a `forEach` with bonus overhead ... – Jaromanda X Aug 08 '16 at 06:06
  • You slay me @T.J.Crowder - now I need to know what is the current best practice when chaining asynchronous tasks together based on an array ... no need for details, just a hint will do me :p – Jaromanda X Aug 08 '16 at 06:11
  • @T.J.Crowder, no offenses taken, but could you expand a little bit more? Apart from the Array.reduce, the rest of the code is wrong? I cannot find another way to do sequential file uploading when these have to be spliced in chunks. How would you do this? – GWorking Aug 08 '16 at 06:14
  • @JaromandaX, you're right, I've got the impression that I had to use Array.reduce for the sequential uploading, maybe I do have to use it but I am doing it wrong here – GWorking Aug 08 '16 at 06:14
  • @Gerard: A couple of things jump out: 1. `Promise.resolve().then(function() { return upload(); })` is basically just `upload()`, as `upload `returns a promise. 2. `Promise.resolve();` when you don't use the return value is a no-op. – T.J. Crowder Aug 08 '16 at 06:17
  • @T.J.Crowder completely right for the 1st, for the 2nd I've got that from https://pouchdb.com/2015/05/18/we-have-a-problem-with-promises.html, but I've probably got it wrong. Getting there however. – GWorking Aug 08 '16 at 06:53
  • @Gerard From what I can gather, upload is supposed to read a file's contents as a string correct? – cdrini Aug 08 '16 at 06:57
  • @Gerard: If you look at that page, you'll note that the return value of `Promise.resolve()` is always used (e.g., it's returned or `.then` is called on it). That's the difference. :-) – T.J. Crowder Aug 08 '16 at 07:00
  • @T.J.Crowder, yes, upload is reading a file but in chunks, so it calls itself until all chunks have been read. And I get your point that if I'm not using the return value, I'm not really using well / understanding well the usefulness of promises. I'll try to re-write the EDITed code in that sense now. – GWorking Aug 08 '16 at 07:09
  • @T.J.Crowder[cont...] Still, looking at the page, when he says "This is also incredibly useful for catching any synchronous errors. It's so useful, that I've gotten in the habit of beginning nearly all of my promise-returning API methods like this:", is he using the returned value? Is he not but he uses it with a very specific purpose and not meant to be widespreadly used? Or is he and I'm not seeing it? – GWorking Aug 08 '16 at 07:10
  • @Gerard If it's reading the file as a string, did you know there's a `reader.readAsText(file)` function? See example [here](http://stackoverflow.com/documentation/javascript/2163/file-api-blobs-and-filereaders/7081/read-file-as-string#t=201608080650355781177) :) – cdrini Aug 08 '16 at 07:12
  • @Gerard: He is: `return Promise.resolve().then(...)`. He's using the return value when he calls `.then` on it. – T.J. Crowder Aug 08 '16 at 07:23
  • @Gerard almost :) You can read the *entire* file using `readAsText`; You don't need to slice at all. – cdrini Aug 08 '16 at 07:27
  • @T.J.Crowder ok, you mean he is not using the return value as a value, but as a "trigger" to say now you go (?). Look at my EDIT2 code, am I also using there the return value with the then (line 14), is that correct? If not, how? I need the then() to be sure I am executing that part when the previous part has been executed – GWorking Aug 08 '16 at 07:31
  • @cdrini I've used now readAsText, thanks!, but what do you mean by not slicing? I can have 3-4 Gb files there, I sliced at the beginning because the browser just crashed – GWorking Aug 08 '16 at 07:32
  • @Gerard Ah, ok, slicing makes sense for larger files. But you're storing the entirety of the contents; won't that make it crash? Usually you slice and handle each chunk so you don't have to store the entire file ([ex](http://stackoverflow.com/documentation/javascript/2163/file-api-blobs-and-filereaders/16626/slice-a-file#t=201608080650355781177)). I'll post some code shortly to clarify what I'm talking about. – cdrini Aug 08 '16 at 07:38
  • @cdrini Well, reading large files makes it crash, storing large data don't. But storing very large data does. For the time being that's not an issue with "my" files, but in the long run I will have to read&process chunks independently and store simplified data if I want to handle larger files. Looking forward for your code, and thousand thanks for your time!!! – GWorking Aug 08 '16 at 07:47
  • @Gerard No problem! Is that how you were using `readAsText` ? Does it still crash? – cdrini Aug 08 '16 at 08:01

1 Answers1

4

You're very close! You need to use the previous value; it should be a promise. Set the initial value of the reduce to be Promise.resolve(). Then inside the reduce function, instead of Promise.resolve().then(...). you should have something like:

return previous
  .then(function() { return upload(current); })
  .then(function() { /* do stuff */ });

It's important that you return here. This will become previous next time the reduce function is called.


You have quite a few issues with the upload function. The biggest issue is that the way you are passing it variables makes it very hard to read :) (and error-prone!)

If you're only reading a text file, use readAsText instead. Note I've renamed it to readFile, because that's a more accurate name.

// returns a promise with the file contents
function readFile(file) {
    return new Promise(function (resolve) {
        var reader = new FileReader();
        reader.onload = function(e) {
            resolve(e.target.result);
        };
        reader.readAsText(file);
    };
}

Then your reduce is simply:

files.reduce(function(previous, file) {
    return previous
      .then(function() { return readFile(file); })
      .then(function(contents) {
          // do stuff
      });
}, Promise.resolve());

You have a big bug with the upload_file variable though. The variable is local to the scope of the reduce function, so it will but undefined inside upload. Pass that in as an argument instead:

function upload(upload_file) { ... }

Side note on var. This is why even though you set upload_file with var inside the reduce function, it would be whatever it was before that function was called for upload:

var a = 3;

function foo() {
  var a = 4;
  console.log(a); // 4
}

foo();
console.log(a); // 3
Graham
  • 7,431
  • 18
  • 59
  • 84
cdrini
  • 989
  • 10
  • 17
  • The link is useful, but it would be much **more** useful if you showed more clearly where this fits into the OP's code, and what parts of the OP's code you're removing. – T.J. Crowder Aug 08 '16 at 06:20
  • Mostly just wanted to paste the link, because I think it fits this case perfectly. Actually, the OPs use of reduce/promises is very close, but upload is quite broken, now that I look at it. That code will result in a lot of issues; posted this too fast. – cdrini Aug 08 '16 at 06:32
  • @cdrini Thanks a lot, working on an updated code, still not working though. With regard to the upload_file, this is defined before the Array.reduce so the scope should be ok (I've made the mistake while adapting the code to the question). However, I do think I should pass it through variable, but before that I'd like to fix/understand this promise barrier – GWorking Aug 08 '16 at 06:55
  • @Gerard your promises are very nearly perfect, it's upload that I'm still trying to wrap my head around :) I was assuming that `upload` returned a promise which resolved when the file was fully uploaded, but the use of variables both inside and outside makes the code difficult to read. If it's defined before the reduce, don't use `var`; this defines it only within the function. – cdrini Aug 08 '16 at 07:06
  • @cdrini I've changed the use of "global" variables for variables that I send to the functions (EDIT2), hopefully clearer to read now? But why don't use `var`? Shouldn't I always(?) – GWorking Aug 08 '16 at 07:36
  • @Gerard updated answer with `readAsText`. You should always use `var`! Need to write code, so extending the answer :) – cdrini Aug 08 '16 at 07:47
  • @cdrini , question: with your `.then(function() { readFile(file); })`, why is it not `.then(function() { return readFile(file); })`? – GWorking Aug 08 '16 at 07:52
  • @Gerard ...because... I was testing you..., to see if you'd catch my completely intentional error... (i.e. whoops! fixed :) ) Added comment about `var` to the answer. – cdrini Aug 08 '16 at 07:54
  • @cdrini That's really fantastic. 48 hours ago I was completely stuck with this, now I can move on and with that feeling that you actually understand something :) Thanks for your time (and the others, of course), marking as accepted answer, hopefully future visitors will appreciate the question. – GWorking Aug 08 '16 at 08:01
  • @Gerard great! Glad it helped :) You can probably remove EDIT/EDIT2 from the question; that will make it a little easier to read for future people. – cdrini Aug 08 '16 at 08:12