How to correctly structure a nested node-fetch request to avoid Promise?

Question

I'm attempting to

Fetch GET my website (with node-fetch)
Scrape it with Cheerio to get specific posts
Fetch GET from my CMS (with node-fetch) to check if there's already a post with the same time
If the check shows no duplicates, Fetch POST into my CMS

The problem I've run into though, is that I can console.log() the duplicate check result, but when I make a conditional with the result for the Fetch POST request, it always returns the check result as a Promise

I'm not sure how to correctly structure my async .then() calls to correctly check.

Code (edited for simplicity):

fetch('https://www.myblog.com/', { method: 'get' })
.then((res) => {
    return res.text()
  })
  .then((data) => {
    const $ = cheerio.load(data)
    const siteHeading = $('.post-item')
    siteHeading.each((index, element) => {
      const title = $(element).find('.entry-title').text()
      const desc = $(element).find('.entry-content').text()
      const exists = fetch(
        'https://mycms.com/api/articles?filters[title][$eq]=' +
          title,
        {
          method: 'GET',
        }
      )
        .then((response) => response.json())
        .then((article) => article.data[0])
        .then((exists) => {
          if (exists) {
            // ^exists always shows up as True, console.log(exists) shows Promise<pending>
            fetch('https://api.broadband.money/api/news-articles', {
              method: 'POST',
              body: JSON.stringify({
                data: {
                  title: title ? title : null,
                  desc: blurb ? blurb : null,
                },
              }),
            })
              .then((response) => response.json())
              .then((data) => console.log(data))
          }
        })
    })
  })

score 0 · Answer 1 · edited Mar 22 '22 at 18:29

0

Using .then() makes code difficult to understand especially when they are nested together. You have to keep track of each of then and take care that the brackets are closed at correct place. Use async await instead. This way you can achieve your objective without so much of nesting: Example:

const data = await fetch('https://www.myblog.com/'{method:'GET'}).then(res=>res.json)

Here the program will wait for the fetch to get completed and then only it will proceed further.

Remember to use async keyword before function declaration inside which you are using fetch.

Example:

async function functionName()
{
..
const data = await fetch...............
..
}

edited Mar 22 '22 at 18:29

Dharman

30,962
25
85
135

answered Mar 22 '22 at 18:23

Marcos

37
7

This doesn't seem to work well for a nested fetch call in a forEach loop. Having an await fetch in it leads to this error: SyntaxError: await is only valid in async functions and the top level bodies of modules – Blue Moon Mar 22 '22 at 21:29
In case of forEach, `async` should be there in function callback of forEach as well. example: ` array.forEach(async(ele)=>{ await fetch..... }) ` – Marcos Mar 23 '22 at 03:27

score 0 · Answer 2 · answered May 07 '22 at 09:04

First, what Marcos advises (combining async/await with .then) is an anti-pattern. Either one or the other must be used.

Second, you use extra .then where you don't need it. Here:

...
.then((exists) => {
...

Thirdly, the forEach method does not wait for the execution of asynchronous code (although in your case this does not seem to matter). Here is a small example for understanding:

function each() {
  console.log("Start");

  const arr = [0, 1, 2, 3];

  arr.forEach(async (el) => {
    await new Promise((resolve) => {
      setTimeout(() => {
        resolve(console.log("This log should be after the start", el));
      }, 1000);
    });
  });

  console.log("Finish");
}

each();

Output:

Start
Finish
This log should be after the start 0
This log should be after the start 1
This log should be after the start 2
This log should be after the start 3

Instead you should use a "for" loop:

async function loop() {
  console.log("Start");

  const arr = [0, 1, 2, 3];

  for (let i = 0; i < arr.length; i++) {
    await new Promise((resolve) => {
      setTimeout(() => {
        resolve(console.log("This log should be after the start", arr[i]));
      }, 1000);
    });
  }

  console.log("Finish");
}

loop();

Output:

Start
This log should be after the start 0
This log should be after the start 1
This log should be after the start 2
This log should be after the start 3
Finish

And finally, it seems to me that replacing this

.then((article) => article.data[0])
.then((exists) => {

with this

.then((article) => {
const exists = article.data[0];

should help. If you provide a working code, I can tell you more.

How to correctly structure a nested node-fetch request to avoid Promise?

2 Answers2