I am trying to pull data from the steam api for dota 2 matches but the api limits the returned matches to 500 total per query. I want to pull as many matches as possible like in the millions so doing it 500 at a time seems like its not gonna happen.
https://wiki.teamfortress.com/wiki/WebAPI/GetMatchHistory
here is some documentation on my request
Basically how it goes is: I make a request to get matches by some parameters, and 500 total are returned per query of which 100 are shown at a time. I then have to update my request url to add the latest match_id and begin my search there. From there the returned matches reduces by 100, so 500 -> 400 -> 300 etc... until 0.
I need way more than 500 but when I refresh my script I get 0 results remaining whenever my last match id is the same as the one in the same 'query'.
Can anyone help me out? My web app idea is 100% dependent on me getting as much games as possible.