I am using a combination of a PhoneGap app and a Django-based backend to record location data points for a research project.
The PhoneGap app will bundle up multiple data points and pushes them to a django-rest-framework API endpoint, for example:
[{
"speed": -1,
"time": 1487292133229.221,
"longitude": 172.123456,
"bearing": -1,
"location_type": "current",
"latitude": -43.123456,
"accuracy": 65,
"heading": -1,
"altitude": 3.585018873214722,
"altitudeAccuracy": 10
}, ...
]
However because of how the PhoneGap app manages location, many of the points are duplicated in the JSON that is pushed.
In order to try and resolve this, I've overridden the ListSerializer to go through the incoming data and compare it to what's already in the database by computing an (unencrypted) unique hash based on timestamp and user id (who uploaded it), and then seeing if something with the same hash already exists. This approach was taken because the actual location data is encrypted before being added to the database.
class DatumListSerializer(serializers.ListSerializer):
def create(self, validated_data):
""" Only append objects if the count of existing objects is 0 (i.e. it is unique),
and it does not appear elsewhere in the list to be committed. """
objects = []
for obj in validated_data:
hashed_value = generate_hash(obj['timestamp'], obj['user'].id)
if Datum.objects.filter(hash=hashed_value).count()==0:
objects.append(Datum.objects.create(user=obj.pop('user').id, **obj))
return objects
This approach manages to catch most of the duplicates, but not all of them. Does anyone have any ideas as to what might be causing non-duplicates to be added? Should I put a unique constraint on the 'hash' field of the Datum model and then silently ignore inserts that cause an IntegrityError?
Thoughts appreciated!
