cant set Host in CURL PHP

Question

I am unable to set the host in curl. It still shows as localhost if i use the following code

function wget($url)
        {

            $agent= 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1';
            $curlHeaders = array (
                    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
                    'Accept-Encoding: gzip, deflate',
                    'Accept-Language: en-US,en;q=0.5',
                    'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1',
                    'Connection: Keep-Alive',
                    'Pragma: no-cache',
                    'Referer: http://example.com/',
                    'Host: hostname',
                    'Cache-Control: no-cache',
                    'Cookie: visid_incap_185989=9v1q8Ar0ToSOja48BRmb8nn1GFUAAAAAQUIPAAAAAABCRWagbDIfmlN9NTrcvrct; incap_ses_108_185989=Z1orY6Bd0z3nGYE2lbJ/AXn1GFUAAAAAmb41m+jMLFCJB1rTIF28Mg==; _ga=GA1.3.637468927.1427699070; _gat=1; frontend=rqg7g9hp2ht788l309m7gk8qi7; _gat_UA-1279175-12=1; __utma=233911437.637468927.1427699070.1427699078.1427699078.1; __utmb=233911437.2.10.1427699078; __utmc=233911437; __utmz=233911437.1427699078.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmt_UA-1279175-1=1; _cb_ls=1; _chartbeat2=S0WVXDwMWnCFBgQp.1427699081322.1427699232786.1; PRUM_EPISODES=s=1427699568560&r=http%3A//example.com/'

            );
            $ch = curl_init();
            curl_setopt ($ch, CURLOPT_HTTPHEADER, $curlHeaders);
            curl_setopt ($ch, CURLOPT_HEADER, TRUE);
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
            curl_setopt($ch, CURLOPT_USERAGENT, $agent);
            curl_setopt($ch, CURLOPT_URL,$url);
            $result=curl_exec($ch);
            return $result;
        }

I use fiddler to track the network requests. where I found the host is still as localhost enter image description here

If I load this same Link in browser i get as following in fiddler

enter image description here

I need my specified domain to be accessed. How can I achieve this? Note: I am aware that host name should not contain the protocol.

Alternatively

Also i would like to know is it possible to get the source code of a website the could be seen in browser through terminal?

I would suggest to start with adding `curl_setopt($ch, CURLOPT_VERBOSE, true);` to your code and checking what is actually happening in the backgound. You will probably get much more information than from fiddler. — Aleksander Wons, Apr 08 '15 at 09:55
I've tried your code. Other file receive header `Host: hostname`. I think you need something like that my website to be masked as "stackoverflow.com", which is difficult to do. You can modify headers, but not HTTP protocol (not easy to do). — Kristiyan, Apr 08 '15 at 10:36
So I ran this on several host I have access to by feeding the url into the the function call and it runs flawlessly. You need to check a few things and provide more information. 1) How is that URL being feed into the function call, are you certain it is set correctly. ie. if you call $result = wget($someurl); Make sure $someurl is set correctly. 2) Check your hostfile and proxy settings. — webternals, Apr 08 '15 at 14:11

Drakes · Answer 1 · 2015-04-09T00:22:20.183

Assuming we are not trying spoof the Host header, omit the Host header altogether and let curl sort it out. In this case, just remove 'Host: hostname', because you already get curl to automatically set this with your code near the bottom with curl_setopt($ch, CURLOPT_URL, $url);.

If you really want to set the Host header yourself, then just replace

'Host: hostname',

with

"Host: ". parse_url($url, PHP_URL_HOST),

(Note: This function doesn't work with relative URLs.)

score 2 · Answer 2 · answered Mar 30 '15 at 09:54

2

try like this,

  curl_init('XXX.XXX.XXX.XXX');
  curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: subdomain.hostname.com'));

answered Mar 30 '15 at 09:54

Ayyanar G

1,545
1
11
24

try like this, $ipaddress='desired_ip'; $url='path/to/file?'; curl_init('http://'.$ipaddress.$url); – Ayyanar G Mar 30 '15 at 12:06
I am working with windows and xampp.. would it be any problem with it? – dharanbro Mar 30 '15 at 12:11
If a single request is sent through this code and then when i try to load this page in browser its asks for captcha – dharanbro Mar 30 '15 at 12:13

score 1 · Answer 3 · edited Apr 07 '15 at 21:20

1

If you are using windows and xampp then try to use virtual host rather than localhost, then it will start working, I did the same.

edited Apr 07 '15 at 21:20

Yanet Salazar Gutierrez

148
1
7

answered Apr 02 '15 at 07:27

Waqas Shakeel

113
7

how to use virtual host – dharanbro Apr 02 '15 at 07:45

score 1 · Answer 4 · answered Apr 08 '15 at 09:14

According to HTTP quick specification read, I assume your problems are happening because of improper Host header being send. I was able to download some websites with following code:

function wget($url, $follow = true) {

    $host = parse_url($url);

    $agent       = 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1';
    $curlHeaders = array(
        'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
        'Accept-Encoding: gzip, deflate',
        'Accept-Language: en-US,en;q=0.5',
        'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1',
        'Connection: Keep-Alive',
        'Pragma: no-cache',
        'Referer: http://example.com/',
        'Host: ' . $host['host'] . (isset($host['port']) ? ':' . $host['port'] : null), // building host header
        'Cache-Control: no-cache',
        'Cookie: visid_incap_185989=9v1q8Ar0ToSOja48BRmb8nn1GFUAAAAAQUIPAAAAAABCRWagbDIfmlN9NTrcvrct; incap_ses_108_185989=Z1orY6Bd0z3nGYE2lbJ/AXn1GFUAAAAAmb41m+jMLFCJB1rTIF28Mg==; _ga=GA1.3.637468927.1427699070; _gat=1; frontend=rqg7g9hp2ht788l309m7gk8qi7; _gat_UA-1279175-12=1; __utma=233911437.637468927.1427699070.1427699078.1427699078.1; __utmb=233911437.2.10.1427699078; __utmc=233911437; __utmz=233911437.1427699078.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmt_UA-1279175-1=1; _cb_ls=1; _chartbeat2=S0WVXDwMWnCFBgQp.1427699081322.1427699232786.1; PRUM_EPISODES=s=1427699568560&r=http%3A//example.com/'
    );
    $ch          = curl_init();
    curl_setopt($ch, CURLOPT_HTTPHEADER, $curlHeaders);
    curl_setopt($ch, CURLOPT_HEADER, TRUE);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, $follow); // following redirects or not
    curl_setopt($ch, CURLOPT_USERAGENT, $agent);
    curl_setopt($ch, CURLOPT_URL, $url);
    $result      = curl_exec($ch);
    return $result;
}

echo(wget('http://example.com'));

Anyway this function is not an universal build. Personally I would add saving cookies between redirection requests etc. Essential change is within 'Host' header line. I'm building there proper Host header based on full $url provided to function.

I found the website that I am trying to scrape is using JS to validate/verify whether the request is sent through browser. if it opened with browser the host is example.com and it loads for a file `example.com/verify.js`. So if the request is sent through browser it loads for `localhost/verify.js`. Before that the above code returns a javascript that i could able to see in browser's inspect element. — dharanbro, Apr 09 '15 at 05:37
@DharanBro Quite not understand your comment. Is it working or it still has some issues? — yergo, Apr 09 '15 at 08:09

score -1 · Answer 5 · edited Apr 08 '15 at 03:16

-1

Set the full URL into CURLOPT_URL.

edited Apr 08 '15 at 03:16

Ethaan

11,291
5
35
45

answered Apr 04 '15 at 08:26

Francisco Félix

2,413
13
16

cant set Host in CURL PHP

5 Answers5