Django how to upload CSV file using Form to populate postgres database and display all items in browser

Question

Django 3.2.1, Python 3.6, Postgres database

EDITED to take into account comments below, thanks!

I am writing a small Django app for storing product information. I coded the backend logic for uploading a local csv file using a Custom Management Command and am connecting this to the front end.

I am having trouble implementing the file upload -> having user upload products.csv via a Form submission to populate the database with file and display all products on one page.

I have stripped down my previous examples, as well as the suggested code below, to the simplest format to try to locate the problem.

Example of the csv file:

name,sku,description
Brian James,skus-look-like-this,The products will have various descriptions. And multiple lines too.

models.py

class Product(models.Model):
    name = models.CharField(max_length=500)
    sku = models.CharField(max_length=500)
    description = models.TextField(blank=False, null=False)
    status = models.TextField(blank=False, null=False, default='inactive')

    class Meta:
        db_table = 'product'

Form for individual product CRUD operations and for CSV file upload.

forms.py

class UploadForm(forms.Form):
    csv_file = forms.FileField(required=False, widget=forms.FileInput(attrs={'class': 'form-control', 'placeholder':
        'Upload "products.csv"', 'help_text': 'Choose a .csv file with products to enter'}))

/templates/upload.html

<form method="post" enctype="multipart/form-data">
        {% csrf_token %}
    <input type="file" name="sent_file" />
    <input type="submit" name="submit" value="Upload" />
</form>

views.py

# Function to upload the form, parse it, save to database
def create_upload(request):
    if request.method == 'GET':
        form = UploadForm()
        return render(request, 'upload.html', {'form': form})

    # If not GET method then proceed
    form = UploadForm(request.POST, request.FILES)
    print('FIRST FORM', form)

    # Validate the form
    if form.is_valid():
            csv_file = form.cleaned_data['csv_file']
            # Errors begin here ^, print(csv_file) = 'None'

            form.save()
            # Crashes here ^ with error: "AttributeError: 'UploadForm' object has no attribute 'save'
"
            file_path = os.path.join(BASE_DIR, form.csv_file.url)
            # printing `file_path` = `AttributeError: 'InMemoryUploadedFile' object has no attribute 'url'
`
        
            # read the file contents and save the product details
            with open(f'{file_path}, r') as products_csv:

            products_file = csv.reader(products_csv)
            next(products_file)  # skip header row

            for counter, line in enumerate(products_file):

                name = line[0]
                sku = line[1]
                description = line[2]

                p = Product()
                p.name = name
                p.sku = sku
                p.description = description
                p.status = random.choice(['active', 'inactive'])
                p.save()

    return redirect('/show_product')
          

Changing form.cleaned_data['csv_file'] to request.FILES['sent_file'] correctly prints the file name uploads.csv but the url is still inaccessible and still crashes on form.save(). The only way I can print to terminal the contents of the uploaded file is by adding this:

csv_file = request.FILES['sent_file']
for i in csv_file:
    print(i)

outputs:

b"'name','sku','description'\n"
b"'Zed','some-skus-more','descriptions. galore.'\n"

But the file still can't be uploaded and form.save() can't be implemented.

I'm not sure how to continue debugging this. If anyone can point me in the right direction, would really appreciate it!

Answer 1

In other to save CSV files you will create a function to read the csv file and save product details: but you can as well refactor the code to meet your suit.

• Upload and save the file first using Product()
• Get the path for the file and read the contents It would be better if you have the same names for model fields and csv columns
• Loop through each line and create a dictionary which contains only a product details at an iteration
• Make an instance of Product() and pass the dictionary to it and save
• For the foreign key, get the object from Product() using get() accordingly with the value that is stored in csv

# You could save the Product details in two ways

new_product = Product()
new_product.registration_number = fields[0]
new_product.customer_name = fields[1] # Change this field name the customer
# like so for other fields

new_product.save()
.....
# Create a model object, create a dictionary of key values where keys corresponds to the field names of the model.

# create a dictionary `new_product_details` containing values of a product

new_product = Product()
new_product.__dict__.update(new_product_details)
new_product.save()

import csv
def save_new_product_from_csv(file_path):
    # do try catch accordingly
    # open csv file, read lines
    with open(file_path, 'r') as fp:
        products = csv.reader(fp, delimiter=',')
        row = 0
        for product in products:
            if row==0:
                headers = product
                row = row + 1
            else:
                # create a dictionary of product details
                new_product_details = {}
                for i in range(len(headers)):
                    new_product_details[headers[i]] = product[i]

                # for the foreign key field you should get the object first and reassign the value to the key
                #We will also have to change the product name to customer_name get the foreign key value
                new_product_details['customer_name'] = Product.objects.get() # get the record according to value which is stored in db and csv file

                # create an instance of product model
                new_product = Product()
                new_product.__dict__.update(new_product_details)
                new_product.save()
                row = row + 1
        fp.close()

Your code should look something like this after:

def uploadcsv(request):
    if request.method == 'GET':
        form = UploadForm()
        return render(request, 'upload.html', {'form':form})

    # If not GET method then proceed
    try:
        form = UploadForm(data=request.POST, files=request.FILES)
        if form.is_valid():
            csv_file = form.cleaned_data['csv_file']
            if not csv_file.customer_name.endswith('.csv'):
                messages.error(request, 'File is not CSV type')
                return redirect('/show_product')
            # If file is too large
            if csv_file.multiple_chunks():
                messages.error(request, 'Uploaded file is too big (%.2f MB)' %(csv_file.size(1000*1000),))
                return redirect('/show_product')

            # save and upload file 
            form.save()

            # get the path of the file saved in the server
            file_path = os.path.join(BASE_DIR, form.csv_file.url)

            # a function to read the file contents and save the product details
            save_new_product_from_csv(file_path)
            # do try catch if necessary
                
    except Exception as e:
        logging.getLogger('error_logger').error('Unable to upload file. ' + repr(e))
        messages.error(request, 'Unable to upload file. ' + repr(e))
    return redirect('/show_product')

Adding a customer name attribute to your model like so:

class Product(models.Model):
    customer_name = models.ForeignKey(settings.AUTH_USER_MODEL,on_delete=models.CASCADE)
    name = models.CharField(max_length=500)
    sku = models.CharField(max_length=500)
    description = models.TextField(blank=False, null=False)
    status = models.TextField(blank=False, null=False, default='inactive')

    class Meta:
        db_table = 'product'

Answer 2

Thanks to this SO post I was able to find an answer by using a generator to decode the CSV line by line.

Here is the code: views.py

def decode_utf8(line_iterator):
    for line in line_iterator:
        yield line.decode('utf-8')


    
def create_upload(request):
    if request.method == 'GET':
        form = UploadForm()
        return render(request, 'upload.html', {'form': form})

    form = UploadForm(request.POST, request.FILES)

    # Validate the form
    if form.is_valid():

        # Get the correct type string instead of byte without reading full file into memory with a generator to decode line by line
        products_file = csv.reader(decode_utf8(request.FILES['sent_file']))
        next(products_file)  # Skip header row

        for counter, line in enumerate(products_file):
            name = line[0]
            sku = line[1]
            description = line[2]

            p = Product()
            p.name = name
            p.sku = sku
            p.description = description
            p.save()

        messages.success(request, 'Saved successfully!')

        return redirect('/show_product')