Prevent adding duplicate data from excel forms imported into db

Question

I have a model on which data imported from an excel files are saved. I want to prevent having duplicate entries,it will check the already existing data to see if they match with the ones .

My model

from django.db import models


class UserData(models.Model):
    GENDER_CHOICES = (
        ('Male', 'Male'),
        ('Female', 'Female'),
    )

    first_name = models.CharField(max_length=50)
    last_name = models.CharField(max_length=50)
    age = models.IntegerField()
    gender = models.CharField(default='Male', choices=GENDER_CHOICES, max_length=6)
    address = models.CharField(max_length=200)

    class Meta:
        verbose_name_plural = 'User Data'

    def __str__(self):
        return self.fullname()

views.py

class UploadFileForm(forms.Form):
    file = forms.FileField()


def import_data(request):
    if request.method == "POST":
        form = UploadFileForm(request.POST,
                          request.FILES)
        if form.is_valid():
            request.FILES['file'].save_to_database(
                name_columns_by_row=2,
                model=UserData,
                mapdict=['first_name', 'last_name', 'age', 'gender', 'address'])


        return HttpResponse("OK")
    else:
        return HttpResponseBadRequest()
else:
    form = UploadFileForm()
return render_to_response('excel/upload_form.html',
                          {'form': form},
                          context_instance=RequestContext(request))

I tried using unique_together and also tried overriding cleaned_data in the form but still can't prevent duplicates being added to the db.

What will be the best way to achieve this? Thanks

At the moment no. But when I tried overriding the cleaned_data I got KeyError saying `first_name` doesn't exist when it clearly does. — Bernard 'Beta Berlin' Parah, Aug 25 '16 at 10:51

score 2 · Accepted Answer · edited May 23 '17 at 12:25

1.Create one model form

class UserDataForm(forms.ModelForm):
    // code here
    class Meta:
        model = UserData

2. make the excel sheet data to dictionary format using.

from pandas import *
xls = ExcelFile('path_to_file.xls')
df = xls.parse(xls.sheet_names[0])

ref: Python Creating Dictionary from excel data.

3.Instantiate the model form with the dictionary data and check for validation. if the data is valid check for the db entry for the same or use django model_name.get_or_create()

form = UploadFileForm(request.POST,
                      request.FILES)
    if form.is_valid():
        # code here to convert to dictionary 
        data = data_in_dict_form
        user_data_form = UserDataForm(data=data)
        if user_data_form.is_valid():
            obj, created = UserData.objects.get_or_create(**user_data_form.cleaned_data)
            # this will create new entry if doesnt exist and just get from db if already exist.
            # You can ause any method to create an entry to the data base. you can also just .filter().exists() to check if there is an existing entry. if It returns True you dont need to save it. 
            # Your Response
         else:
            # your Response

Note: If there are multiple rows , each rows are for separate entries in db. you need to loop each rows and validate and save()/Do_not_save. :)

I hope this would help you to get the problem resolved.

May be I am missing something obvious but still can't figure out **data_in_dict_form** in `data = data_in_dict_form`. Can this be clarified? — Love Putin Not War, Jun 04 '20 at 17:08

Prevent adding duplicate data from excel forms imported into db

1 Answers1