Using yql and php to scrape content

Question

been fooling around with YQL trying to understand it a little better. I've managed to pull the info I want from an external site and get a results node in the YQL console but have thus been unsuccessful in displaying the results on my local development server.

What I ultimately want to do is try and put this into a wordpress function so I can call it on a page (for example the standings page).

The code I used in php (edit:: I changed the code to this)

ini_set('display_errors', 1);
    ini_set('log_errors', 1);
    error_reporting(E_ALL);

    $yql_base_url ="http://query.yahooapis.com/v1/public/yql";
    $yql_query = 'SELECT * FROM html WHERE url="http://www.nwjhl.com/leagues/standingsTotals.cfm?leagueID=15654&clientID=4594" AND xpath=\'//table[@class="yuiboxscores"]/tbody/tr\'';

    $yql_query_url = $yql_base_url . "?q=" . urlencode($yql_query);
    $yql_query_url .= "&format=json";

    // set up the cURL
    $c = curl_init();
    curl_setopt($c, CURLOPT_URL, $yql_query_url);
    curl_setopt($c, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($c, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($c, CURLOPT_SSL_VERIFYHOST, false);

    // execute the cURL
    $rawdata = curl_exec($c);
    curl_close($c);

    // Convert the returned JSON to a PHP object
    $data = json_decode($rawdata);

I've managed to get the data Im looking for now Im just having trouble pulling the data in as Im assuming its because the "tr" is in an array.

Ive been probably staring at its probably a little to complex for figuring with YQL for me, I figure a for loop is needed, I just unsure of how to reference the [content] and then of course all the other numbers. I tried to debug with

$results = $data->query->results->td;
    echo '<pre>' . print_r($results) . '</pre>';

Im not sure how I would be able to loop through all these objects a arrays to display [content] (which is the team) and then of course there statistics. Continuing to work on this hopefully I can figure out

I'm pretty sure you can a) skip the 'select' portion of this query, and b) see your query output (in either json or XML) using the yahoo query console at https://developer.yahoo.com/yql/console/. You'll still do the curl request, but the console will give you the URL to fetch as well. It's likely marginally faster than your solution, because Yahoo will already have parsed your 'select' query into a directly fetchable URL. The other nice thing about the console is that you get to see exactly what's returned, including field-names. I think it would have answered your question here handily. — mdg, Mar 19 '15 at 02:24

Andrew Edvalson · Accepted Answer · 2012-03-23T21:01:01.613

The json you are getting back from that URL is not what you expect it to be. Try

echo $output

right after curl_close($ch);

then running it through a json formatter (google it). $data->query->results works, but there is no child called Result, it just seems to be a jsonified html table of some sort.

Once you see the JSON in a more human readable format your problem might make more sense to you.

Per your Edit - Try this after you json_decode the raw data. I'm not sure exactly what pieces of data you are after, but I'm guessing it will be in the objects you are dumping with var_dump

$topics = $data->query->results->table->tbody->tr;
foreach ($topics as $topic)
   {
   echo "row:";
   var_dump($topic);
   }

Here's a more specific example of how to loop through this and find

elements

$topics = $data->query->results->table->tbody->tr;
foreach ($topics as $topic)
   {
   $data = $topic->td;
   foreach ($data as $element)
      {
      if (array_key_exists('a', $element))
         {
         echo "Team Name: " . $element->a->content . "\n";
         }
      if (array_key_exists('p', $element))
         {
         echo "Found P: " . $element->p . "\n";
         }
      }
   }

Thanks, yeah I see my errors, I made an edit to my original post, the hardest part for me now is trying to get the necessary information like [content] and each TD's [p] with the stats displayed through PHP. — Phrosty30, Mar 23 '12 at 09:00
Thanks Andrew, that is the data Im after with one other thing, Im trying to get the [content] which is essentially the name. I've tried $data->a and then running something similar in code above but is returning error, I dont have a problem with things like [width] which are in the array but a has two options, href and content. Basically Im after Name [Content] then there will be 14 values (wins, losses etc) for each Name, then Im going to try and loop it to move to the next record. Thanks for your help! — Phrosty30, Mar 23 '12 at 18:08

Using yql and php to scrape content

1 Answers1