crawler with ramda.js (functional programming)

Question

I'm trying to crawl movie data from TMDB website. I finished my code with pure javascript, but I want to change the code into functional programming style by using ramda.js.

I attached my code below. I want to get rid of for-loop (if it is possible) and use R.pipe function.

(async () => {
  for (let i = 0; i < 1000; i++) {
    (() => {
      setTimeout(async () => {
        let year = startYr + Math.floor(i / 5);
        await request.get(path(year, i % 5 + 1), async (err, res, data) => {
          const $ = cheerio.load(data);
          let list = $('.results_poster_card .poster.card .info .flex a');
          _.forEach(list, (element, index) => {
            listJSON.push({
              MovieID: $(element).attr('id').replace('movie_', ''),
              Rank: (i % 5) * 20 + index + 1,
              Year: year
            });
          });
          if(i === 1000 - 1) {
            await pWriteFile(`${outputPath}/movieList.json`, JSON.stringify(listJSON, null, 2));
          }
        });
      }, 1000 * i);
    })(i);
  }
})().catch(error => console.log(error));

TMDB has api, not really need to crawl it: https://www.themoviedb.org/documentation/api — Hitmands, Jul 19 '19 at 07:02
I appreciate for your comment, I'm just trying to do that for studying. — Hyeon, Jul 19 '19 at 13:33

score 0 · Answer 1 · edited Jul 18 '19 at 18:02

0

You can use the Ramda range() function to replace your loop.

https://ramdajs.com/docs/#range

R.range(0, 1000);

That will provide you with a collection of integers (your i) that you can work with (map() or whatever you need).

edited Jul 18 '19 at 18:02

customcommander

17,580
5
58
84

answered Jul 18 '19 at 17:48

Tom Faltesek

2,768
1
19
30

Edgard Leal · Accepted Answer · 2019-07-26T17:42:41.560

Steps:

1- Break your code in small functions
2- Stop using async await and use promise.then(otherFunction)
3- When using promise, you could create a sleep function like these: const sleep = (time) => new Promise(resolve => setTimeout(resolve, time));

Ex.:

const process = index => sleep(1000)
   .then(() => makeRequest(index))
   .then(processData);

R.range(0, 1000)
   .reduce(
       (prev, actual) => prev.then(() => process(actual),
       Promise.resolve()
   ) // Sequential
   .then(printResult);

R.range(0, 1000)
   .map(process) // Parallel
   .then(printResult);

crawler with ramda.js (functional programming)

2 Answers2