1

I have built a web scraper that gets multiple URLs at once collects data and pushes it to a MongoDB. The total number of urls is 400k plus and it collects data of 100 urls at once and then uploads it to MongoDB atlas in a single collection, but after exactly 7399 documents it throws a maximum call stack size exceeded. And after I restart the loop from 7399 it then stops at 58998 exactly.

The Code

      successCount += success.length;
      failedCount += failed.length;
      // FeedDataToMedData(success, failed, "drugs");
      console.log(success.length, failed.length);
      if (success.length != 0) {
        //   await (await DB())
        //     .collection("drugs-success")
        //     .insertMany(success, { ordered: false });

        const toInsert = success.map((item) => ({
          insertOne: { ...item },
        }));
        await (await DB()).collection("drugs-success").bulkWrite(toInsert, {
          ordered: false,
        });
      }
      if (failed.length != 0) {
        //   await (await DB())
        //     .collection("drugs-failed")
        //     .insertMany(failed, { ordered: false });
        const toInsert = failed.map((item) => ({
          insertOne: { ...item },
        }));
        await (await DB()).collection("drugs-failed").bulkWrite(toInsert, {
          ordered: false,
        });
      }
    }

The error

RangeError: Maximum call stack size exceeded
    at calculateElement (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:29:26)
    at calculateObjectSize (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:22:28)
    at calculateElement (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:158:21)
    at calculateObjectSize (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:22:28)
    at calculateElement (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:158:21)
    at calculateObjectSize (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:22:28)
    at calculateElement (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:158:21)
    at calculateObjectSize (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:22:28)
    at calculateElement (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:158:21)
    at calculateObjectSize (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:22:28)
RangeError: Maximum call stack size exceeded
    at calculateElement (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:29:26)
    at calculateObjectSize (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:22:28)
    at calculateElement (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:158:21)
    at calculateObjectSize (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:22:28)
    at calculateElement (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:158:21)
    at calculateObjectSize (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:22:28)
    at calculateElement (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:158:21)
    at calculateObjectSize (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:22:28)
    at calculateElement (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:158:21)
    at calculateObjectSize (/Users/eshansingh/dev/farmako/1mg-scraper/node_modules/bson/lib/parser/calculate_size.js:22:28)
Eshan Singh
  • 11
  • 1
  • You have a loop in your data structure. As you are writing a web scraper, probably some link in the site points to a page you already scraped but you don't recognize that correctly ... – derpirscher Sep 16 '22 at 21:57
  • yes, I am scraping a website, no I had run a validation check that prevents similar links from being repeated. So that should not happen. – Eshan Singh Sep 17 '22 at 07:24

0 Answers0