-1

I am scraping data from a crowdfunding music based website. for research purposes. I am specifically trying to get data for the projects under the 'Rock' genre on the website, however when I get the source code using my web crawler, it is different from what I can see if I "view-source" on my browser when analyzing the website, and the data I need is nowhere to be found.

I've inspected the website and have tried to determine the source of the data with no success. Is this a common problem when scraping websites?

The web crawler I am using is written in VBA and has already been used to scrape data from other websites, this is the first time I encounter something like this.

Sigma
  • 123
  • 1
  • 2
  • 8
  • You need to wait for Javascript to run – SLaks Dec 07 '15 at 18:35
  • Need to show some code from your web crawler - are you just fetching the source, or are you automating IE to load the page? It's not unusual for a page to update it own content after it's loaded, so you will need to account for that. – Tim Williams Dec 07 '15 at 19:04
  • @TimWilliams I am fetching the source via my web crawler. I did the same with previous websites that I successfully crawled. I simply store the source file in a string. I am aware that some pages may update their content after being loaded. If that was the case I wouldn't be able to see the source code by right clicking on the website and viewing its source code. – Sigma Dec 07 '15 at 20:47
  • @SLaks how would I do that? – Sigma Dec 07 '15 at 20:47

1 Answers1

0

The data you are trying to access must be generating from JavaScript on that page and that is why you cant see it in your view source. To make it a counter check, in chrome you can do 'inspect' and then go to view source. compare the data of the same page,the data generated by JavaScript will be in inspect window but not in view source.

Umer Qureshi
  • 1