How To Capture All Intermediate URL in a Redirect and Export Them Into a CSV File

Question

I have a list of URL using multiple redirection like this:

url1=>url1redirect1=>url1redirect2=>url1redirect3= >url1final url2=>url2redirect1=>url2redirect2=>url2final ...

The list is in this format:

url1

url2

url3

I don't own all the website in the redirection chain. Some of them are third party tracking software.

Is there a way to capture all intermediary urls and the final url and export them into a neat csv file like this:

url1,url2,url3,

url1redirect1,url2redirect1,url3redirect1,

url1redirect2,url2redirect2,url3redirect2,

url1redirect3,url2final,url3redirect3,

url1final,,url3redirect4,

...

where do you have this list on urls, in a file ? can you provide the code you tried ? — Ghilas BELHADJ, Aug 05 '13 at 06:05

score 0 · Answer 1 · answered Aug 05 '13 at 07:07

I've found this function called get_all_redirects that can do the job:

function get_redirect_url($url){
    $redirect_url = null; 

    $url_parts = @parse_url($url);
    if (!$url_parts) return false;
    if (!isset($url_parts['host'])) return false; //can't process relative URLs
    if (!isset($url_parts['path'])) $url_parts['path'] = '/';

    $sock = fsockopen($url_parts['host'], (isset($url_parts['port']) ? (int)$url_parts['port'] : 80), $errno, $errstr, 30);
    if (!$sock) return false;

    $request = "HEAD " . $url_parts['path'] . (isset($url_parts['query']) ? '?'.$url_parts['query'] : '') . " HTTP/1.1\r\n"; 
    $request .= 'Host: ' . $url_parts['host'] . "\r\n"; 
    $request .= "Connection: Close\r\n\r\n"; 
    fwrite($sock, $request);
    $response = '';
    while(!feof($sock)) $response .= fread($sock, 8192);
    fclose($sock);

    if (preg_match('/^Location: (.+?)$/m', $response, $matches)){
        if ( substr($matches[1], 0, 1) == "/" )
            return $url_parts['scheme'] . "://" . $url_parts['host'] . trim($matches[1]);
        else
            return trim($matches[1]);

    } else {
        return false;
    }

}

function get_all_redirects($url){
    $redirects = array();
    while ($newurl = get_redirect_url($url)){
        if (in_array($newurl, $redirects)){
            break;
        }
        $redirects[] = $newurl;
        $url = $newurl;
    }
    return $redirects;
}

You can use it that way:

$urls = file_get_contents("urls.txt");

$url_list = explode("\n", $urls);

$file_content = '';
foreach ($url_list as $url){
    $rez = get_all_redirects($url);
    $file_content .= "$url,";
    foreach ($rez as $v){
        $file_content .= "$v,";
    }
    $file_content = substr($file_content,0, -1);
    $file_content .= "\n";
}

file_put_contents("output.csv", $file_content);

urls.txt is a text file containing your urls (one url in each line):

http://url1.com
http://url2.com
http://url3.com
...
http://urlN.com

How To Capture All Intermediate URL in a Redirect and Export Them Into a CSV File

1 Answers1