Synonym of [web-scraping]
Synonym of web-scraping: Let's [scrape] these tags off the bottom of our shoe
Questions tagged [scraper]
349 questions
-2
votes
1 answer
Build a Sinatra server that displays the current job postings when I visit that server’s root path
I currently have a scraper file called scraper.rb. I need to figure out how to take the output from this and have it display on a Sinatra server. If you could also provide an explanation of why your answer works that would be great, thanks in…
HiStakes
- 17
- 1
-2
votes
1 answer
How to make my function run every hour?
With the help of some online tuts (Bucky), I've managed to write a simple web scraper that just checks if some text is on a webpage. What I would like to do however, is have the code run every hour. I assume I will need to host the code also so it…
user1663396
-2
votes
1 answer
Getting all pdf files from a domain (for example *.adomain.com)
I need to download all pdf files from a certain domain. There are about 6000 pdf on that domain and most of them don't have an html link (either they have removed the link or they never put one in the first place).
I know there are about 6000 files…
Panagiotis Panagi
- 9,927
- 7
- 55
- 103
-3
votes
1 answer
output/rendered text can not be seen in source code
As all Facebook social plugins have this feature;
Your Facebook name can be seen on the web page but when you look up in the source code you can not see Facebook name.
So I need to know why and how?
This feature may be used in order to avoid…
jwin27
- 1
- 1
-3
votes
1 answer
Python MIT Course OCW Web Page SCraper
So I got this MIT Scraper program to get it working. Somebody worked on it before and has been told that it's functioning and the coding is correct. I just have to fix some config issue and should be written.
First of all here is the link to the…
-3
votes
1 answer
Scrape data from website from on change event using node.js
I'm trying scrape data from one website. In that when page load there is drop down list and I have to select specific value from the dropdown.
For scrapping data from web I'm using cheerio reference link is https://www.npmjs.com/package/cheerio.…
Sachin Shah
- 4,503
- 3
- 23
- 50
-3
votes
1 answer
how to get all elements (visible and invisible)
i stuck in list view where thousands of items but only 20 items show when i scroll down then 20 items more show and so on...
- item1
- item2
- item3 . . .
- item1000…
Rashid Aziz
- 36
- 7
-3
votes
1 answer
Python Link Scraper
focus_Search = raw_input("Focus Search ")
url = "https://www.google.com/search?q="
res = requests.get(url + focus_Search)
print("You Just Searched")
res_String = res.text
#Now I must get ALL the sections of code that…
-3
votes
1 answer
Facebook Profile Picture scraper using Graph API v2.5 and access token
I am building Facebook profile picture scraper and using Phasher class to convert scraped pictures to Hexadecimal values and store it inside the database to compare it for similar pictures, Now I was using this http request to fetch for the pictures…
Jadolyo
- 39
- 8
-3
votes
1 answer
web scraping with python to snatch a file
Hi I want to snatch csv file in the URL please see below [download].
Being new to python i gotten this far can someone leverage what i have. many thanks.
from requests import session
import bs4
payload = {
'action': 'login',
'username':…
Ricard Le
- 55
- 1
- 2
- 12
-3
votes
1 answer
Scraping a web page: Javascript?
I want to extract some data from an HTML page.
I tried it with php, but I got an issue because this page is only available if you are connected to a specific network: unfortunately, my client is connected to that network, but not my server, so php…
user1071138
- 656
- 3
- 12
- 30
-3
votes
2 answers
perl regex for a string
I have a string from where I need to extract street , city , state , zip .
The string may look like
a)$str1 ="2500 South 3850 West Suite A Salt Lake City, UT 84120-7225";
b)$str2 ="19701 DaVinci Lake Forest, CA 92610";
c)$str3="abc…
jnanchak
- 1
- 5
-3
votes
3 answers
is it possible to make that scraper act in extra pages when the webpage have it?
from twill.commands import *
from bs4 import BeautifulSoup
from urllib import urlopen
import urllib2
with open('urls.txt') as inf:
urls = (line.strip() for line in inf)
for url in urls:
try:
urllib2.urlopen(url)
…
muchacho
- 55
- 1
- 6
-4
votes
2 answers
Data screaping based on Search engines
Is it possible to scrap the web based on Keywords using Search engines in PHP?
Like when some put keyword, the script will search google and render the results and then render the pages and scrap/extract the line that includes the matched…
user9774304
-4
votes
2 answers
PERL WWW::Scripter not work, not return content
Usualy I make scrapers in Ruby, but decide to do in Perl. And when I run my script I see number of url which opens very very very slow.
And I thank, maybe its redirect problem? Or maybe its JS urls thats why problem. And I decide to use some module…
rogersnest
- 1
- 2