PHP variables in scraper function

Question

I'm using ScraperWiki to build a simple screen scraper getting links from an online store. The store has multiple pages, so I want to get all the links from the first page, find the "next" button in the pager, go to that url, find all the links from there, go to next page, and so on and so forth.

Here's where I'm at. The ScraperWiki uses Simple HTML DOM and CSS selectors:

<?php 
require 'scraperwiki/simple_html_dom.php';  

function nextPage(){
    $next = $html->find("li.pager-next a"); 
    $nextUrl = 'http://www.domain.com';
    $nextUrl .= $next->href . "\n"; 
    getLinks($nextUrl);
} 

function getLinks($url){    // gets links from product list page   

    $html_content = scraperwiki::scrape($url);
    $html = str_get_html($html_content);

    $x = 0;

    foreach ($html->find("div.views-row a.imagecache-product_list") as $el) { 
        $url = $el->href . "\n";  
        $allLinks[$x] = 'http://www.domain.com';
        $allLinks[$x] .= $url;
        $x++;
    }

    nextPage();
}

getLinks("http://www.domain.com/foo/bar");


print_r($allLinks);

?>

The getLinks() function works fine when NOT in a function, but I'm getting "undeclared variable" errors when I put them in a function. My question is:

In PHP can I declare empty variables/arrays to use throughout the script, like in Javascript? I've read a few answers here on Stack which seems to imply that there is no need to declare, which seems odd.

Variable scope still exists. Either use parameters, or invite "`global`" variables into each function. — mario, Feb 21 '13 at 23:28
possible duplicate of [Undefined variable error when calling a function from another snippet](http://stackoverflow.com/questions/14301958/undefined-variable-error-when-calling-a-function-from-another-snippet) — mario, Feb 21 '13 at 23:30
@mario Cheers for that, couldn't find a good reference. Here the issue is that before the function is called the variables don't have value, so I can't declare them. What's the best way to pass them through in the example above? — JVG, Feb 21 '13 at 23:34

score 1 · Answer 1 · answered Feb 21 '13 at 23:34

1

If you had shown the whole error it would probably be something like

Undefined variable: $getLinks

Probably because what you meant was this: getLinks($nextUrl);

Not this: $getLinks($nextUrl);

It works fine outside of the nextPage function because you are calling it correctly there.

answered Feb 21 '13 at 23:34

Nick Pickering

3,095
3
29
50

Nope, I'm getting: `PHP Notice: Undefined variable: allLinks in /home/scriptrunner/script.php on line 33` when I try to `print_r($allLinks);` at the end of the script. – JVG Feb 21 '13 at 23:44
That means $allLinks is undefined. Define it as an empty array. If that `for` loop never runs, then $allLinks never gets defined. – Nick Pickering Feb 21 '13 at 23:48
use `$allLinks = array();` just after your include and I think you'll be set. Your scraper isn't returning anything. – Nick Pickering Feb 21 '13 at 23:55

score 0 · Answer 2 · answered Feb 21 '13 at 23:38

0

class ScraperWiki{
    public $variable;
    protected $variable;
    private $variable;

    // here you have the option of choosing how your functions and variables are treated...


     private function getLinks(){...}



     public function someOtherFunction(){
          $this->getLinks(); //will call the function in this Class
     }
}

Plus you have a syntax Error $getLinks($nextUrl); should be getLinks($nextUrl)

answered Feb 21 '13 at 23:38

Drmjo

584
1
5
21

He's building it procedurally, he had no intention of turning it into a class. – Nick Pickering Feb 21 '13 at 23:39
building a class is not always best practice. If you don't need the features of OOP, then you're adding overhead for no reason. – Nick Pickering Feb 21 '13 at 23:45

score 0 · Answer 3 · answered Feb 22 '13 at 22:14

Found the solution myself with a bit of help from other answers - had to declare $allLinks at the start of the script, outside of any functions. In Javascript that would be enough to make it global, but in PHP it looks like you have to also declare it as global INSIDE functions, like this:

$allLinks = array();

function foo(){
    global $allLinks
    ...//stuff
}

This finally got my code working.

PHP variables in scraper function

3 Answers3