I recently upgraded to MacOS Ventura and discovered that software I'd written in the past to scrape data from a webpage would no longer work. That code used initWithContentsOfURL to access and download a sequence of pages which my code would then process (yes, I know synchronous networking is bad practice, but this is a research app only I use, not commercial software).
I then modified to code to use NSURLSessionTask and dataTaskWithURL to grab the pages asynchronously, which worked great, except... since my software is now making 10 requests in rapid order, the website is generating Error 1015 informing me that I'm being rate limited and temporarily banned.
I guess this means I now have to slow down my code when making the requests? My code looks like this (cutting it down from 10 requests to 3, removing error handling, and NSURL initialization):
NSURLSession *session = [NSURLSession sharedSession];
__block NSURLSessionTask *task1 = nil;
__block NSURLSessionTask *task2 = nil;
__block NSURLSessionTask *task3 = nil;
task1 = [session dataTaskWithURL:pageURL1 completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
if (data) {
task1Source = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
}
[task2 resume];
}];
task2 = [session dataTaskWithURL:pageURL2 completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
if (data) {
task2Source = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
}
[task3 resume];
}];
task3 = [session dataTaskWithURL:pageURL3 completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {
if (data) {
task3Source = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
}
}];
[task1 resume];
The site I'm hitting obviously does not like this, which has generated the rate limit error. Is there a preferred way to distribute the requests or otherwise generate an acceptable delay that will be acceptable to the site (I also have a request out to the site's tech support)?
Thanks!