-2

I'm new to Scrapy. I have Googled around and searched in Stack Overflow, but there's no exact things I want to do. I have been struggling with these for two days.

This is what I have gotten so far for pipelines.py. Would anyone point out what's wrong with it or show me some example code for connecting Scrapy to MySQLdb using Peewee?

from MySQLdb import *
from peewee import *


mysql_db = MySQLDatabase('nasdaq_db', user='root', passwd='')

class Quote(Model):
    """A base model that will use our MySQL database"""
    time = CharField()
    price = CharField()
    volume = CharField()
    class Meta:
        database = mysql_db


db.connect()
Quote.create_table()

class RealTimeQuotePipeline(object):
    def process_item(self, item, spider):
        item = Quote(time=item['time'], price=item['price'], volume=['volume'])
        item.save()

Run:

scrapy crawl nasdaq

Error Message:

peewee.OperationalError: (1049, "Unknown database 'nasdaq_db'")

If I change it to:

mysql_db = MySQLDatabase(db='nasdaq_db', user='root', passwd='')

There is another error message:

TypeError: __init__() takes at least 2 arguments (1 given)
halfer
  • 19,824
  • 17
  • 99
  • 186
Ken Hui
  • 31
  • 3
  • I am not a Python dev, but I wonder if you are trying to tackle too much at once. Scrapy (a scraper) and Peewee (an ORM) are presumably very separate. So, narrow things down - do the scrape first, with no ORM, and get that working. Once that is OK, connect it to Peewee. Can you use that approach to make the question a bit more specific as to what you are stuck on? – halfer May 01 '15 at 18:56
  • (I don't know the reason for the DVs, but don't worry about them. If you can make the question more focussed, someone may upvote it! It is not productive to ask why people have voted in the way they have, and commentary of that kind definitely does not belong in questions). – halfer May 01 '15 at 18:58
  • 1
    Although you are not a Python dev, your suggestion and edition of my questions are really helpful. Thank you, halfer. – Ken Hui May 02 '15 at 00:07

1 Answers1

2

You need to be sure the database nasdaq_db exists in your mysql instance. You can open up mysql and run:

create database nasdaq_db;

Then you should be able to connect. You had the correct syntax the first time around:

db = MySQLDatabase('nasdaq_db', user='root', passwd='')
coleifer
  • 24,887
  • 6
  • 60
  • 75
  • Thank you, coleifer. I fixed the problems after create database. I just realized that peewee won't create mysql database automatically. It's a helpful answer! – Ken Hui May 02 '15 at 00:05