0

I have recently run into an issue I cannot solve using Axios and Cheerio. What I am trying to do is scrape a coffee store's webpage that contains multiple different coffee products. So far I have been able to scrape each of the the coffee names, countries, its flavors, and the urls to each single product page. What I would like to do is take it a step further and scrape each of the single product urls and return its stock availability. On each single product page there is a description div that contains some text saying whether it is in stock or not. However I have run into the issue of scraping each url for that text.

So far this is what I have:

const links = [];

  axios(coffeeStoreURL)
  .then(response => {
      const html = response.data;
      const coffeePageHTML = cheerio.load(html);
      const coffees = [];

      coffeePageHTML('.coffeeBadge', html).each(function() {
          const coffeeName = coffeePageHTML(this).find('.coffeeBadgeBottom a').text()
          const coffeeCountry = coffeePageHTML(this).find('.country-label').text()
          const coffeeFlavors = coffeePageHTML(this).find('.flavors').text()
          const coffeeImage = coffeePageHTML(this).find('img').attr('src')

          const coffeeURL = coffeePageHTML(this).find('a').attr('href')
          

          coffees.push({
              coffeeImage,
              coffeeName,
              coffeeCountry,
              coffeeFlavors,
              coffeeURL
          });

          links.push((
            coffeeURL
          ))

      });
  })
  .then( () => {
    for (let i = 0; i < links.length; i++) {
      axios(links[i])

      //This is where I am stuck on how to scrape each link in my links array and use cheerio to grab the div data from each page

    }

  }).catch(err => console.log(err));

I assume I am on the right path but if not then any guidance would be much appreciated.

Thank You

m0nja
  • 1

1 Answers1

0

The quick but less elegant way to do this is by calling a self-executing anonymous function (AKA an IIFE) that holds each iteration inside your for loop.

for (let i = 0; i < links.length; i++) {
  (function(link) {
    // do your axios calls here, "link" is holding the current URL
  })(links[i]);
}

If you want something more elegant then check out this question: Axios.get().then() in a for loop

JM-AGMS
  • 1,670
  • 1
  • 15
  • 33