Stack Overflow
Stack Overflow

Questions tagged [scraper]

Synonym of [web-scraping]

Synonym of : Let's [scrape] these tags off the bottom of our shoe

349 questions
2
votes
0 answers

Flipkart Review Scraping - Data not coming after certain pages, why?

I was trying to build a crawler for Flipkart using python and bs4. I was successful in doing so as well, but there is a problem while trying to scrape more than 13-14 pages of review. Till 13 pages the scraper works fine, but once I reach page no.…
Prateek
  • 185
  • 1
  • 3
  • 12
2
votes
1 answer

missing a piece of the "set-Cookie" from a GET response (scraper)?

I'm trying to authenticate with: https://sso-prod.sun.ac.za/cas/login but the cookies I receive on my initial get request seem to be incomplete here is what firefox receives: but the cookies I get from my request: Cookies: "" …
SimpleJack
  • 151
  • 1
  • 12
2
votes
2 answers

How can I extract text from a span tag using beautiful soup 4?

how can I scrape text through span tags using beautful soup? scrape faculty members informations from bs4 import BeautifulSoup import requests r = requests.get("http://www.uoj.ac.ae/ContentBan.aspx?m=15&p=4&sm=4") soup = BeautifulSoup(r.content,…
Kt7
  • 71
  • 1
  • 4
2
votes
1 answer

Selenium webdriver with python to scrape dynamic page cannot find element

So there are a lot of questions that have been asked around dynamic content scraping on stackoverflow, and I went through all of these, but all the solutions suggested did not work for the following problem: Context: Using Selenium webdriver with…
Audrey Bascoul
  • 23
  • 4
2
votes
1 answer

Scrape Google News with lxml and python

I'm trying to scrape Google News using python and lxml. Everything is going well but when I try to print each div data using a for loop everything mess up. Here my code: # -*- coding: utf-8 -*- from stem import Signal from stem.control import…
JJack_
  • 859
  • 9
  • 30
2
votes
1 answer

What Python tools can I use to write a scraper of a password-protected webpage?

Suppose there is a password-protected website that I want to access to scrape some info from it and put it into a spreadsheet. For example, it could be my personal credit card account page and I would be scraping info about the latest transactions.…
I Z
  • 5,719
  • 19
  • 53
  • 100
2
votes
1 answer

How to Parse this HTML with Web::Scraper?

I am trying to use Web::Scraper to parse the following HTML:

TITLE1
DESCRIPTION1

TITLE2
DESCRIPTION2

TITLE3
DESCRIPTION3

into 'test' => [ …
user1768233
  • 1,409
  • 3
  • 20
  • 28
2
votes
1 answer

Web Scraper won't fill in a child window that my VBA code launches

I have the following code in VBA which opens up an IE page, fills it in and then clicks on a button that opens up a new IE window. However, my code is not able to fill in the first dropdown of the new window. Any help would be greatly…
chris
  • 155
  • 2
  • 3
  • 17
2
votes
1 answer

Scraping .aspx page in node

I've been working on scraping the following site: http://www.fightingillini.com/schedule.aspx?path=softball I've had extensive experience using node/cheerio/scraperjs to scrape both static and dynamic content in the past, but I'm not having any luck…
Mark
  • 51
  • 1
  • 9
2
votes
0 answers

Can DOMPath be used to get javascript variables inside HTML document?

I want to get javascript variable printed out in the HTML code using DOMXPath,here is the HTML code