1

I am building a scraping tool with the Laravel Goutte Package https://github.com/FriendsOfPHP/Goutte I have been able to scrape most websites until I came across this website http://www.bhutanpost.bt/ which I need to scrape.

The problem I am suspecting is that the site has charset as UTF-7 and the returned xml is not the same as it is shown in "view source". The elements I am trying to scrape do exist in the view source so I can say they are not pulled dynamically by JS.

Any help will be highly appreciaetd.

penjor
  • 21
  • 3
  • Well, what error messages are you getting? – ceejayoz Feb 08 '21 at 16:58
  • no error as such, in other sites i get the exact HTML as in the view source page, but on this site I only get this: +ADw?xml version+AD0AIg-1.0+ACI encoding+AD0AIg-UTF-7+ACI standalone+AD0AIg-yes+ACI?+AD4\n.... and a few more such lines [doesnt fit here] – penjor Feb 08 '21 at 17:03

1 Answers1

1

I dug through it and found a dirty fix, The problem was with the loadHtml function of DomCrawler which was inside parseXhtml function. When the meta tag is not explicitly defined loadHtml causes problems so here was my fix:

$dom->loadHTML('<meta http-equiv="Content-Type" content="text/html; charset='.$charset.'">'.$htmlContent);

I prepended the meta details before the HTML content.

penjor
  • 21
  • 3