0

I am building a web scraper and I am new to it. What it does is that it scrapes the amazon prices for the URL provided by the user which then sends an email whenever the price goes below the price set by the user. I am using SendGrid to send the emails.

However, I had a question. I wanted to set the script to run automatically every hour or so and send an email if the price is found to be less. After researching online, I found that node-cron can help me do that and it did. However, it only works when my script is running in the background.

So my main question is that to run the script and send the emails, does my script need to actually run the entire time? If not what can I do?

This is my cron.js file. I also have a AmazonTracker.js which has code for tracking the price and sending the email.

const cron = require('node-cron');
let shell = require('shelljs');

const url = process.argv[2]
// minimum price for which user wants an alert
const minPrice = process.argv[3]

const emailID = process.argv[4];

// scheduled to run every second
cron.schedule("* * * * * *", function() {
  console.log("running");
  if(shell.exec(`node AmazonTracker.js ${url} ${minPrice} ${emailID}`).code !== 0){
    console.log("Something went wrong");
  }
})

Also, I get this error when trying to run the code for some links. My script works for some links and throws this error for some others. If anyone has any suggestions.

(node:4284) UnhandledPromiseRejectionWarning: Error: .wait() for #priceblock_dealprice timed out after 30000msec
    at newDone (C:\Users\dtdan\node_modules\nightmare\lib\actions.js:545:9)
    at Timeout._onTimeout (C:\Users\dtdan\node_modules\nightmare\lib\actions.js:578:5)
    at listOnTimeout (internal/timers.js:549:17)
    at processTimers (internal/timers.js:492:7)
TanDev
  • 359
  • 3
  • 5
  • 13

1 Answers1

0

Yes, the logic behind Crons is, that some daemon process will keep looking at the time and trigger some task. since you are using node-cron you need to keep this piece of code running it will trigger your AmazonTracker.js,

ideally one should go with systems Cron, which is more reliable and prone to failure due to system shutdown etc.

check below (this is one of the examples) https://www.cyberciti.biz/faq/how-do-i-add-jobs-to-cron-under-linux-or-unix-oses/

r7r
  • 1,440
  • 1
  • 11
  • 19
  • Ohh I see. Is there a way that my script automatically fires without manually running it and keeping it running in the background. Thanks – TanDev Aug 04 '20 at 08:42
  • Sorry, I am a windows user. – TanDev Aug 04 '20 at 08:47
  • there are other options, you can have docker on windows, and inside docker have your scheduler on Linux image, you can also use windows task scheduler, check below link https://joshuatz.com/posts/2020/using-windows-task-scheduler-to-automate-nodejs-scripts/ Google it you will find many options to schedule node using windows scheduler – r7r Aug 04 '20 at 09:18