-1

For a personal research project, I will need a large dataset of League of Legends game histories of a large number of players (at least the results of 20 games for about 1000 players).

I first looked at the Riot Games API. I asked for a personal key but the limitations of the allowed queries are not enough. A production key could do the trick but they are not provided for personal projects of this type.

So I thought I'd get around it by scrapping data from sites like OP.GG. However, when you load a URL on this site, the recent data (which I'm interested in) are not updated and you have to click on the "update" button, then wait a while. I think scrapping data via this site would take too long (I would need to simulate browser sessions for each page).

I imagine many people have already tried to get data like this. How did you get around this problem?

Mikael Dúi Bolinder
  • 2,080
  • 2
  • 19
  • 44
Poco
  • 9
  • 3
  • you can try to use threads or multiprocessing to run many browsers at the same time. But it can make another problem - server may have system to detect many requests from one IP in short time and block this IP to stop bots/scripts/spamers/hackers. And this may need to use Proxy Servers to connect with different IPs. But free proxy servers usually are outdated (they don't work or servers already block them). And maybe API will be simpler (even if it needs few hours to get it). But maybe if you ask other person to register and get second key then you could run two scripts at the same time. – furas Mar 16 '22 at 15:38
  • with API you are also safe - they don't block your access to server. With scraping they can block you and you get nothing. – furas Mar 16 '22 at 15:44

1 Answers1

1

The development keys and the personal keys have a rate limit of 100 requests per 2 minutes.

You need to look up 1000 players, then 1000 match histories, 20 000 matches and 20 000 match timelines. This adds up to 42 000 requests.

42000/50 = 840. 840/60 = 14. So it'd just take you about half a day to do it the legal way, instead of endless tweaks on a scraper.

Mikael Dúi Bolinder
  • 2,080
  • 2
  • 19
  • 44