Access data of a wiki scraper and store it in a local database

Question

Hi i have written a scraper on scraper-wiki which scrape a web page and store the data in scraper-wiki database.Now i want to write a program in python which will go to scraper-wiki API and fetch the data and stores in a sq lite database of local machine.

score 0 · Answer 1 · answered Apr 20 '13 at 06:26

0

First, you need to query the data you want. Here is the documentation: 1 2

Then, you need to store it using the sqlite library which its documentation can be found at python's official documentation site.

answered Apr 20 '13 at 06:26

Ghost93

175
1
2
12

thanks for your answer @Ghost but my requirement is little different The python program should run on local machine not on scraper wiki as a second scraper. – Binit Singh Apr 20 '13 at 06:49
Maybe I didn't fully understand. You have a scraper, and you want to query it and put it to a local sqlite db, don't you? The program running on a local machine queries the local db or the scraper? – Ghost93 Apr 20 '13 at 07:09
yes i have a scraper and i want to query the swdata of that scrapper from a python program running on my local machine and store the results in local db – Binit Singh Apr 20 '13 at 07:19
But the documentation you are referring to is saying about the methods to attch one scrapper within another scrapper with the command scraperwiki.sqlite.attach("new_americ_foundation_drone_strikes", "src") print scraperwiki.sqlite.table_info("src.swdata") – Binit Singh Apr 20 '13 at 09:05
Check the _Direct SQL for selecting_ section. `print scraperwiki.sqlite.select("* from ttt") [{u'yy': u'hello', u'xx': 9}, {u'yy': u'again', u'xx': 10}] ` – Ghost93 Apr 20 '13 at 09:39
scraperwiki.sqlite.select() it dosen't work in command prompt of your system – Binit Singh Apr 20 '13 at 10:09

score 0 · Accepted Answer · answered Apr 22 '13 at 05:29

Hey guys luckily i got the answer On scraper wiki there is an API here https://scraperwiki.com/docs/api#sqlite which will generate the URI of the table of a particular wiki scraper u want access of then u can open it as a url with the help of urllib module of python

import urllib

def access_data():

f= urllib.urlopen('https://api.scraperwiki.com/api/1.0/datastore/sqlite?format=csv&name=example&query=select%20*%20from%20%60table%60')

data=f.read()
s = open('example.csv','w')
s.write(data)

score 0 · Answer 3 · answered May 06 '13 at 13:09

0

There's also a download link on your scraper's overview page where you can download your data as a sqlite file, no code required. Then you can work with the data locally.

answered May 06 '13 at 13:09

Ross

406
2
4

Access data of a wiki scraper and store it in a local database

3 Answers3