Parsing only text content from url

Question

I am trying to parse text content from url given. Here is the code:

<?php
$url = 'http://stackoverflow.com/questions/12097352/how-can-i-parse-dynamic-content-from-a-web-page';
$content = file_get_contents($url);
echo $content;                          // This parse everything on the page, including image + everything

$text=escapeshellarg(strip_tags($content));
echo "</br>";
echo $text;   // This gives source code also, not only the text content over page
?>

I want to get only the text written over page. No page source code. Any idea for this? I already googled but above method only present everywhere.

@CORRUPT: I have already used it. It also gives source code. Not only the text written on page. — user123, Sep 22 '13 at 11:10
I am passing entire content as a argument to c++ program. `escapeshellarg` quote the result by `''` so that it can be encountered as single argument — user123, Sep 22 '13 at 11:17

jaudette · Accepted Answer · 2013-09-22T16:30:46.323

4

You can use DOMDocument and DOMNode

$doc = new DOMDocument();
$doc->loadHTMLFile($url);
$xpath = new DOMXPath($doc);
foreach($xpath->query("//script") as $script) {
    $script->parentNode->removeChild($script);
}
$textContent = $doc->textContent; //inherited from DOMNode

Instead of using xpath, you can also do:

$doc = new DOMDocument();
$doc->loadHTMLFile($url); // Load the HTML
foreach($doc->getElementsByTagName('script') as $script) { // for all scripts
    $script->parentNode->removeChild($script); // remove script and content 
                                               // so it will not appear in text
}
$textContent = $doc->textContent; //inherited from DOMNode, get the text.

edited Sep 22 '13 at 16:30

answered Sep 22 '13 at 11:11

jaudette

2,305
1
20
20

thanks but still we get ` "https:") { p += "s"; d = "engine"; } var z = document.createElement("script"); z.type = "text/javascript"; z.async = true; z.src = p + "://" + d + ".adzerk.net/ados.js"; var s = document.getElementsByTagName("script")[0]; s.parentNode.insertBefore(z, s); var ados = ados || {}; ados.run = ados.run || []; ados.run.push(function () { ados_setKeywords('java,javascript,html-parsing,jsoup,dynamic-data');; ados_load(); }); var _gaq=_gaq||` etc etc – user123 Sep 22 '13 at 11:25
1

@Karimkhan It is text content of ` – dev-null-dweller Sep 22 '13 at 11:32
I have to keep it generalized, because user may enter any url! – user123 Sep 22 '13 at 11:35
@jaudette: Why it gives bunch of warnings? ` arning: DOMDocument::loadHTMLFile(): htmlParseEntityRef: expecting ';' in http://stackoverflow.com/questions/12097352/how-can-i-parse-dynamic-content-from-a-web-page, line: 255 in /opt/lampp/htdocs/FB/ec2/test.php on line 13 Warning: DOMDocument::loadHTMLFile(): htmlParseEntityRef: expecting ';' in http://stackoverflow.com/questions/12097352/how-can-i-parse-dynamic-content-from-a-web-page, line: 255 in /opt/lampp/htdocs/FB/ec2/test.php on line 13 Warning: DOMDocument::loadHTMLFile(): htmlParseEntityRef: expecting ';' in htt ` etc etc – user123 Sep 22 '13 at 12:36
@Karimkhan Is it the same warnings for every url, or specific to your page? What is on line 13? According to doc: `While malformed HTML should load successfully, this function may generate E_WARNING errors when it encounters bad markup. libxml's error handling functions may be used to handle these errors.` – jaudette Sep 22 '13 at 13:38
Those warning got removed by 'libxml_use_internal_errors(true);' see http://stackoverflow.com/questions/18943954/removing-docdocument-warnigng-while-parsing-page-content – user123 Sep 22 '13 at 16:20

score 1 · Answer 2 · answered Sep 22 '13 at 11:07

1

$content = file_get_contents(strip_tags($url));

This will remove the HTML tags coming form the page

answered Sep 22 '13 at 11:07

Moeed Farooqui

3,604
1
18
23

score 1 · Answer 3 · answered Sep 22 '13 at 11:08

1

To remove html tag use:

$text = strip_tags($text);

answered Sep 22 '13 at 11:08

MD SHAHIDUL ISLAM

14,325
6
82
89

score 1 · Answer 4 · answered Sep 22 '13 at 11:14

1

A simple cURL will solve the issue. [TESTED]

<?php
$ch = curl_init("http://stackoverflow.com/questions/12097352/how-can-i-parse-dynamic-content-from-a-web-page");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); //Sorry forgot to add this
echo strip_tags(curl_exec($ch));
curl_close($ch);
?>

answered Sep 22 '13 at 11:14

Shankar Narayana Damodaran

68,075
43
96
126

thanks but still it gives code like ` $('#tell-me-more').click(function () { var clickSource = $("body").attr("class") + '-mini'; if ($("body").hasClass("questions-page")) { clickSource = 'questionpagemini'; } if ($("body").hasClass("home-page")) { clickSource = 'homepagemini'; } StackExchange.using("gps", function () { StackExchange.gps.track("aboutpage.click", { aboutclick_location: clickSource } , ` etc. Which should not come – user123 Sep 22 '13 at 11:21
You need to do a lot of parsing to get the data. Before that you need to get permission from SO as you are using their resources without using any API as this is synonymous to 'Leeching' – Shankar Narayana Damodaran Sep 22 '13 at 11:32
its not only about so, any url can be there! – user123 Sep 22 '13 at 11:35

Parsing only text content from url

4 Answers4