populate text file to database

Question

I'm trying to read from a text file that contains a bunch of words line by line. I tried to use this question's solution but I couldn't do it. This is how I setup my models.py file:

from django.db import models
class words(models.Model):
    wordslist = models.CharField(null='True', max_length=128)

I tried to run this code in the python manage.py shell:

from app.models import words
WORDLIST_FILENAME = "app/words.txt"
inFile = open(WORDLIST_FILENAME, 'r')
wordlist = []
for line in inFile: 
    wordlist.append(line.strip().lower()) # wordlist: list of strings
    words.objects.create()

This doesn't end up completing and I have to keyboard interrupt. I'm using Django 1.11

How large is the file? Also your not closing (or you didn't post it) the file explicitly with `close` or implicitly `with open file()`, your leaking memory. If the file is large you could be creating a massive array. — reticentroot, Jul 07 '17 at 17:56
Adding on to what I said earlier.. your `words.objects.create()` is also not doing anything useful, you're not passing data into the create function. The link you provide gives you an example of that `Representative.objects.create(**dict(zip(fields, row)))` — reticentroot, Jul 07 '17 at 18:21
file size is 633KB, i'm using `words.objects.create(wordslist=line)` and `with open(file_path)` — Binarydreamer, Jul 09 '17 at 02:29
So definitely not memory. Can you provide an example of the text file so that we can run local and reproduce the issue. — reticentroot, Jul 09 '17 at 02:46
I got it to stop infinitely looping but when I call `Words.objects.all()` returns `, ...]` instead of the actual words, [text file is in problem set 3](https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-0001-introduction-to-computer-science-and-programming-in-python-fall-2016/assignments/) — Binarydreamer, Jul 09 '17 at 03:20

score 1 · Answer 1 · answered Jul 07 '17 at 18:22

Here is a working example of pulling lines out of a file as storing them in the database using your words model which should be capitalized.

I would add this to the model since you might do this sometime in the future with more files. Potentially you could get a list of file paths and run a for loop, passing the file paths into this function to do the work.

def lines_in_file_to_db(file_path):
    """ Handles reading lines from a file and saving to the Database.

    :param file_path: Path to where file is located.
    :type file_path: str
    """
    data = []

    with open(file_path) as file:
        for line in file:
            # do line manipulation in here...
            line = line.replace('\n', '')
            data.append(line)

    for line in data:
        words.objects.create(wordslist=line)

Sorry for the delayed response, I tried to fix the problem while implementing your solution, but when I run this code `from play.models import Words def lines_in_file_to_db(file_path): data = [] with open(file_path) as file: for line in file: data.append(line.strip().lower()) for line in data: Words.objects.create(wordslist=line) lines_in_file_to_db('play/words.txt')` it just hangs untill keyboard interrupt — Binarydreamer, Jul 09 '17 at 02:15

populate text file to database

1 Answers1