PHP cURL - how to emulate exactly same request like user?

Question

I am trying to make a website scraper, but the website is acting diferrently, than normal request via browser.

How can i make perfect cURL reguest, that the website will not filter it and block it?

Any help would be appriciated.

$curl_handle = curl_init ("***");
$header = array();
$header[] = "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0";
$header[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$header[] = "Accept-Language: cs,en-US;q=0.7,en;q=0.3";
$header[] = "Accept-Encoding: utf-8";
$header[] = "Connection: keep-alive";
$header[] = "Host: ****";


curl_setopt($curl_handle, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0');
curl_setopt($curl_handle, CURLOPT_HTTPHEADER, $header);
curl_setopt ($curl_handle, CURLOPT_COOKIEFILE, dirname(__FILE__) . '/cookie.txt');
curl_setopt ($curl_handle, CURLOPT_COOKIEJAR, dirname(__FILE__) . '/cookie.txt');
curl_setopt ($curl_handle, CURLOPT_RETURNTRANSFER, true);
curl_setopt ($curl_handle, CURLOPT_AUTOREFERER, true);

$output = curl_exec ($curl_handle);

This is, what i got so far, but it is still getting blocked.

How the website is acting,what is your code and what you want to achieve? — Crunch Much, Oct 25 '15 at 15:14
When i request the page via browser, the response is ok, but when i request via cURL, it redirects me to the homepage, so i think, it is kind of blocking mine scraper. — Tadeáš Jílek, Oct 25 '15 at 15:15
Why not simply open up your chrome, and visit the website with console tools open and capture the request and response headers right there? — Jesse, Oct 25 '15 at 15:21
Yeah, i did this, those headers are headers, what i have caught like this. — Tadeáš Jílek, Oct 25 '15 at 15:26

score 0 · Answer 1 · answered Nov 16 '16 at 20:35

The following CURL options might help:

curl_setopt($ch, CURLOPT_REFERER, $_SERVER['REQUEST_URI']);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

PHP cURL - how to emulate exactly same request like user?

1 Answers1