How can I make this PHP script run faster/asynchronously?

Question

I have a pastebin scraper script, which is designed to find leaked emails and passwords, to make a website like HaveIBeenPwned.

Here is what my script is doing:
- Scraping Pastebin links from https://psbdmp.ws/dumps
- Getting a random proxy using this Random Proxy API (because Pastebin bans your IP if you hammer too many requests): https://api.getproxylist.com/proxy
- Doing a CURL request to the Pastebin links, then doing a preg_match_all to find all the email addresses and passwords in the format email:password.

The actual script seems to be working alright, but it isn't optimized enough, and is giving me a 524 timeout error after some time, which I suspect is because of all those CURL requests.

Here is my code:
api.php

    function comboScrape_CURL($url) {
    // Get random proxy
    $proxies->json = file_get_contents("https://api.getproxylist.com/proxy");
    $proxies->decoded = json_decode($proxies->json);
    $proxy = $proxies->decoded->ip.':'.$proxies->decoded->port;
    list($ip,$port) = explode(':', $proxy);

    // Crawl with proxy
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL,$url);
    curl_setopt($ch, CURLOPT_PROXY, $proxy);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_HEADER, 1);
    $curl_scraped_page = curl_exec($ch);
    curl_close($ch);
    comboScrape('email:pass',$curl_scraped_page);
}

index.php

require('api.php');
$expression = "/(?:https\:\/\/pastebin\.com\/\w+)/";

$extension = ['','1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16','17','18','19','20'];
foreach($extension as $pge_number) {
    $dumps = file_get_contents("https://psbdmp.ws/dumps/".$pge_number);
    preg_match_all($expression,$dumps,$urls);
    $codes = str_replace('https://pastebin.com/','',$urls[0]);
    foreach ($codes as $code) {
        comboScrape_CURL("https://pastebin.com/raw/".$code);
    }
}

`because Pastebin bans your IP if you hammer too many requests` PS, you can buy a Pastebin.com Pro account, and you get access to a scraping that isn't limited. — hanshenrik, Nov 24 '18 at 11:30
I've already tried opening pastebin with a proxy, and it works, but as soon as I try using the proxy API is stops. — Mark Adewale, Nov 24 '18 at 11:31
if you buy a pastebin.com pro account, you get access to the scraping api ( https://pastebin.com/doc_scraping_api ) which can fetch about 86400 pastes per day without getting ip banned. — hanshenrik, Nov 24 '18 at 11:34

score 0 · Accepted Answer · answered Nov 24 '18 at 11:43

524 timeout error - err, seems you're running php behind a web server (apache? nginx? lighthttpd? IIS?) don't do that, run your code from php-cli instead, php-cli can run indefinitely and never timeout.

because Pastebin bans your IP if you hammer too many requests - buy a pastebin.com pro account instead ( https://pastebin.com/pro ), it costs about $50 (or $20 around Christmas & Black Friday), and is a lifetime account with a 1-time payment, and gives you access to the scraping api ( https://pastebin.com/doc_scraping_api ), with the scraping api you can fetch about 1 paste per second, or 86400 pastes per day, without getting ip banned.

and because of pastebin.com's rate limits, there is no need to do this asynchronously with multiple connections (it's possible, but not worth the hassle. if you actually needed to do that however, you'd have to use the curl_multi API)

Thanks for the help, ended up buying PRO in the end :) – Mark Adewale Nov 24 '18 at 11:58 — Mark Adewale, Nov 24 '18 at 11:58

How can I make this PHP script run faster/asynchronously?

1 Answers1