1

Please go easy on me, I literally just started using Django at 3am this morning...

As part of a school project, I wrote a Python script to scrape data, generate SQLite3 tables and insert records from a Python dictionary into said tables. The schema of this SQLite3 db looks like this:

CREATE TABLE state (
                        id INT PRIMARY KEY,
                        name VARCHAR(100) NOT NULL
                      );
CREATE TABLE city (
                        id INT PRIMARY KEY,
                        name VARCHAR(100) NOT NULL
                      );
CREATE TABLE restaurant (
                        id INT PRIMARY KEY,
                        name VARCHAR(100) NOT NULL,
                        rating FLOAT,
                        url VARCHAR(300),
                        price VARCHAR(4),
                        review_count INT,
                        street VARCHAR(100),
                        city VARCHAR(100),
                        state VARCHAR(2),
                        country VARCHAR(100),
                        zip_code INT,
                        phone VARCHAR(20),
                        image_url VARCHAR(300)
                      );
CREATE TABLE category (
                        id INT PRIMARY KEY,
                        title VARCHAR(100)
                      );
CREATE TABLE restaurant_by_category (
                        restaurant_id INT,
                        category_id INT,
                        FOREIGN KEY (restaurant_id) REFERENCES restaurant,
                        FOREIGN KEY (category_id) REFERENCES category
                      );
CREATE TABLE review (
                        url VARCHAR(300) PRIMARY KEY,
                        restaurant_id INT,
                        rating FLOAT,
                        name VARCHAR(100),
                        time VARCHAR(100),
                        text VARCHAR(300)
                      );

Weeks after making this database, I then decided to start making the web app using Django. I have a (basically) fully operational Django app with a SQLite3 backend whose models.py looks like this:

class City(models.Model):
  name = models.CharField(max_length=100)

  def __str__(self):
    return self.name


class State(models.Model):
  name = models.CharField(max_length=100)

  def __str__(self):
    return self.name


class Restaurant(models.Model):
  name = models.CharField(max_length=100)
  rating = models.DecimalField(max_digits=2, decimal_places=1, null=True,
      blank=True)
  url = models.CharField(max_length=300, blank=True, null=True)
  price = models.CharField(max_length=4)
  review_count = models.IntegerField(blank=True, null=True)
  phone = models.CharField(max_length=14, blank=True, null=True)
  image_url = models.CharField(max_length=300, blank=True, null=True)
  city = models.ForeignKey(City, blank=True, null=True)
  state = models.ForeignKey(State, blank=True, null=True)
  state_abbreviation = models.CharField(max_length=2, blank=True, null=True)
  zip_code = USZipCodeField(blank=True, null=True)
  street = models.CharField(max_length=100, blank=True, null=True)

  def __str__(self):
    return self.name


class Review(models.Model):
  url = models.CharField(max_length=300)
  restaurant_id = models.IntegerField(blank=True, null=True)
  rating = models.DecimalField(max_digits=2, decimal_places=1, null=True,
      blank=True)
  name = models.CharField(max_length=100, blank=True, null=True)
  time = models.DateTimeField(blank=True, null=True)
  text = models.CharField(max_length=100, blank=True, null=True)


class Category(models.Model):
  title = models.CharField(max_length=100, blank=True, null=True)


class RestaurantByCategory(models.Model):
  restaurant_id = models.ForeignKey(Restaurant)
  category_id = models.ForeignKey(Category)

My preexisting SQLite3 database and my Django models are almost exactly the same except for these two differences:

  • The Restaurant model has an additional field state_abbreviation which is actually state in the SQLite3 database -- state in the Restaurant model does not have a corollary field in the SQlite3 database. When I migrate the SQLite3 database over to Django, I don't care if the state field in the Django model is None because there are only three states in the database and I can easily add that in manually via the admin page, but I do want the state attributes in the SQLite3 database to be entered into the state_abbreviation fields of my Restaurant model instances.

  • There are several field type mismatches such as the DateTimeField in Review, the USZipCodeField in Restaurant, or the ForeignKey in Restaurant...all of these are declared as VARCHAR in my SQLite3 database

My question is, how can I automatically migrate my existing SQLite3 database into my new Django models quickly and safely?

Here is what I've researched:

Here is why I'm stuck and can't figure out what to do next:

  • The later option (fixtures) doesn't seem like what I'm looking for because, from what I understand, this would only work if I was migrating to a different database from another database where both the new and the old databases are in my Django app...my situation requires dumping data from an SQLite schema which I don't think is possible -- at least not in a way that solves my problem.
  • The former option seems kinda like what I want, except the example makes it seem as though I'd have to manually write model instances for all the records anyway which wouldn't be any better than just entering all the records in through the admin page.

  • I also feel like the field type mismatches and slightly differing schema (see the differences of state/state_abbreviation attributes in Restaurant) would throw off this migration process even if there was a way to automatically migrate from my SQLite3 database to my Django models.

So, am I pretty much stuck with entering in records manually or is there an easy way to perform this migration? There are only about 400 records in the SQLite3 table, so I could manually enter a bunch in to get a working demo before presenting this project tomorrow, but I also have a lot of other things to do before this is finished and I'd rather not spend 3 hours manually entering records.

Thanks in advance for the help.

aweeeezy
  • 806
  • 1
  • 9
  • 22
  • Additional options: Rewrite the models in Django and [copy the data from the old database](https://stackoverflow.com/questions/29220677/sqlite-easiest-way-to-copy-a-table-from-one-database-to-another/29221750#29221750); or use the database as-is and [disable Django's model management for the models](https://docs.djangoproject.com/en/1.11/ref/models/options/#managed) – Colonel Thirty Two May 08 '17 at 23:25
  • Thanks @ColonelThirtyTwo -- the first link + plus a migration file did the trick for me. – aweeeezy May 10 '17 at 09:01
  • @ColonelThirtyTwo I think that should really be an answer (with a little more detail of course) – e4c5 May 13 '17 at 05:23
  • if he doesn't answer, please add your own answer explaining what you did. Because this is a good question – e4c5 May 13 '17 at 05:24

0 Answers0