I am doing a data extraction project where i am required to build a web scraping program written using python using selenium and phantomjs headless webkit as browser for scaping public information like friendlist in facebook.The program is starting fairly fast but after a day of running it is getting slower and slower and I cannot figure out why ?? Can anyone give me an idea why it is getting slower ? I am running on a local machine which pretty good specs of 4gb ram and quad core processor . Does FB provide any API to find friends of friends ?
Asked
Active
Viewed 209 times
3
-
1It could be because of your program consuming too much memory space. – SuperAzeem Aug 12 '14 at 02:52
-
is there a way to check for sure that it is comsuming large memory space ?? – Soumya Aug 13 '14 at 05:27
-
You should post the code for us to see. – SuperAzeem Aug 13 '14 at 07:43
-
I have basically used a BFS search – Soumya Aug 19 '14 at 04:20
1 Answers
0
We faced the same issue. We resolved this by closing browser automatically after particular time interval. Clear temporary cache and open new browser instance and continue the process.
QAMate.com
- 141
- 1
- 3
-
does quiting a webdriver instance of phantomjs automatically deletes cache content ? – Soumya Aug 19 '14 at 04:06
-
Its supposed to delete cache, but its an open source software, we can not expect quality all the time. WebDriver's core is browser automation. Rest we have to take care. – QAMate.com Aug 22 '14 at 03:45