ES7 async/await conceptual issue

Question

I am migrating an existing program to use async/await (via Babel's bluebirdCoroutines) in order to learn this style. I have been looking at this tutorial.

I am a bit troubled by the following behavior. This snippet works as expected:

let parts = [];
let urlsP = urls.map((url, index) => { 
    return dlPart(url, index, tempDir); 
});
for (let urlP of urlsP) { // Parallel (yay!)
    parts.push(await urlP);
}
for (let part of parts) { // Sequential
    await appendFile(leFile, part);
}

Re-written as follows, it still works but the fist operation is not parallel any more (it takes much longer to finish)!

let index = 0;
let parts = [];
for (let url of urls) { // NOT Parallel any more!!!
    index++;
    parts.push(await dlPart(url, index, tempDir));
}
for (let part of parts) {
    await appendFile(leFile, part);
}

This is the implementation of dlPart()

function dlPart(url, num, dir) {
    var cmd = 'wget --quiet "' + url + '" -O ' + dir + "/" + num;
    return exec(cmd).then(() => {
        return dir + '/' + num;
    });
}

What am I missing?

Answer 1

The reason that it is no longer running in parallel is because of the timing of when you are creating your promises in the two examples. This is described above comment more clearly.

In your first example you start all the Promises which begin to execute their functions. Then in this loop:

for (let urlP of urlsP) { // Parallel (yay!)
    parts.push(await urlP);
}

You wait for the first promise to be done, and then the second promise to be done, etc. But the whole time you are waiting for the first promise to be done ALL the other promises are still executing. Hence they run in "parallel".

In your second example, you both START and AWAIT the promises inside the loop instead of starting them all before the loop. So in this code:

for (let url of urls) { // NOT Parallel any more!!!
    index++;
    parts.push(await dlPart(url, index, tempDir));
}

The parts.push line does the following in order:

1. Runs dlPart() which returns a promise and starts downloading the part
2. Waits for the promise to resolve
3. Pushes the resolved value into parts.

All the other promises have not been started and are not running "in parallel" they only get started when it's their turn in the loop. This means they get called one at a time, and only start the next one executing once the previous one is done and that is why they run iteratively.

Note: .map is not asynchronous if it were then your first example wouldn't work with large lists because the for of loop would start before all the promises were added to your urlsP array.

Answer 2

The .map function is asynchronous so the rest of your code doesn't have to wait on it, it will finish when ready. Then you replaced it with a for loop that holds everything back while it completes.

Answer 3

You can better see the differences between the code when its written in a slightly different way.

For all intents and purposes this is exactly what Sam has explained, but in a way that I find helps developers understand it in a way that they are more accustomed to.

ES7 Code

let parts = [];
let urlsP = urls.map((url, index) => { 
    return dlPart(url, index, tempDir); 
});
for (let urlP of urlsP) { // Parallel (yay!)
    parts.push(await urlP);
}

ES6 Code

let parts = [];
// Capture all the pending promises into an array
let urlsP = urls.map((url,index)=>{
    // Returns the promise to the urlsP array
    return dlPart(url,index,tempDir);
});
// Catch all the data in an array
Promise.all(urlsP).then(res=>{
    parts=res;
});

To reiterate what Sam has explained the post above. In the ES7 example the map function calls of all the async functions and creates a new array of promises. The for of loop iterates through the array of promises and checks to see if the promise has been resolved yet, if it hasn't it will wait until the that specific promise resolves, then repeat this process. If you were able to watch this code in slow motion with a debugger tag in chrome you would notice that some promises will be resolved by the time the loop checks to see if it was resolved, while others you will have to wait for

The ES6 example is essentially the same, with the only difference in how we get the parts array. In this case the Promise all response is an array of all the resolved values, thus we make parts equal to the response instead of pushing into the array

---

Now imagine writing the following code in es6:

ES7 Code

let index = 0; let parts = []; 
for (let url of urls) { // NOT Parallel any more!!!
    index++;
    parts.push(await dlPart(url, index, tempDir)); 
}

You would have to use a generator or a recursive function, My understanding of generators is still pretty new so I will show a recursive function

ES5/6 Code

let index = 0; let parts = []; let promises = [];

function getParts(index,){
      return new Promise((resolve,reject)=>{
            dlPart(urls[index],index,tempDir).then(res=>{
                parts.push(res)
                if(index<urls.length-1){
                    promises.push(getParts(++index));
                    resolve()
                }else{
                    resolve()
                }
            }
      }
    }
promises.push(getParts(index));
Promise.all(promises).then(finished=>{
    // Then you can continue with whatever code
});

Now with this ES7 example code above you will notice that the for of loop iterates through the urls array and waits for the promise to resolve before moving to the next index of the array.

The ES6 example does the same in the sense that it will begin with the url at index 0, waits for dlPart promise to resolve, pushes the response into the parts array, checkes that the index is still smaller than the urls array length, getParts then calls itself again, until finally it runs out of urls indices and resolves its last promise so that the code below the Promise.all(promises) may begin to run

When you begin looking at the differences in the readability between ES6 and ES7 you can see why async/await were finalized in the es7 spec.