How to replace Django's primary key with a different integer that is unique for that table

Question

I have a Django web application that uses the default auto-incremented positive integers as the primary key. This key is used throughout the application and is frequently inserted into the URL. I don't want to expose this number to the public so that they can guess the number of users or other entities in my Database.

This is a frequent requirement and I have seen questions to similar mine with answers. Most solutions recommend hashing the original primary key value. However, none of those answers fit my need exactly. These are my requirements:

1. I would like to keep the Primary Key field type as Integer.
2. I also would prefer not to have to hash/unhash this value every time it is read or written or compared to the database. That seems wastefuly It would be nice to do it just once: When the record is initially inserted into the Database
3. The hashing/encryption function need not be reversible since I don't need to recover the original sequential key. The hashed value just needs to be unique.
4. The hashed value needs to be unique ONLY for that table -- not universally unique.
5. The hashed value should be as short as possible. I would like to avoid extremely long (20+ characters) URLs

What is the best way to do achieve this? Would the following work?

def hash_function(int):
    return fancy-hash-function # What function should I use??


def obfuscate_pk(sender, instance, created, **kwargs):
    if created:
        logger.info("MyClass #%s, created with created=%s: %s" % (instance.pk, created, instance))
        instance.pk = hash_function(instance.pk)
        instance.save()
        logger.info("\tNew Pk=%s" % instance.pk)

class MyClass(models.Model):
    blahblah = models.CharField(max_length=50, null=False, blank=False,)


post_save.connect(obfuscate_pk, sender=MyClass)

Answer 1

The Idea

I would recommend to you the same approach that is used by Instagram. Their requirements seems to closely follow yours.

Generated IDs should be sortable by time (so a list of photo IDs, for example, could be sorted without fetching more information about the photos) IDs should ideally be 64 bits (for smaller indexes, and better storage in systems like Redis) The system should introduce as few new ‘moving parts’ as possible—a large part of how we’ve been able to scale Instagram with very few engineers is by choosing simple, easy-to-understand solutions that we trust.

They came up with a system that has 41 bits based on the timestamp, 13 o the database shard and 10 for an auto increment portion. Sincce you don't appear to be using shards. You can just have 41 bits for a time based copmonent and 23 bits chosen at random. That does produce an extremely unlikely 1 in 8.3 million chance of getting a conflict if you insert records at the same time. But in practice you are never likely to hit this. Right so how about some code:

Generating IDs

START_TIME = a constant that represents a unix timestamp

def make_id():
    '''
    inspired by http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram
        '''
    
    t = int(time.time()*1000) - START_TIME
    u = random.SystemRandom().getrandbits(23)
    id = (t << 23 ) | u
    
    return id


def reverse_id(id):
    t  = id >> 23
    return t + START_TIME 

Note, START_TIME in the above code is some arbitary starting time. You can use time.time()*1000 , get the value and set that as START_TIME

Notice that the reverse_id method I have posted allows you to find out at which time the record was created. If you need to keep track of that information you can do so without having to add another field for it! So your primary key is actually saving your storage rather than increasing it!

The Model

Now this is what your model would look like.

class MyClass(models.Model):
   id = models.BigIntegerField(default = fields.make_id, primary_key=True)  

If you make changes to your database outside django you would need to create the equivalent of make_id as an sql function

As a foot note. This is somewhat like the approach used by Mongodb to generate it's _ID for each object.

Answer 2

You need to separate two concerns:

1. The primary key, currently an auto-incrementing integer, is the best choice for a simple, relatively predictable unique identifier that can be enforced on the database level.

2. That does not mean you have to expose it to users in your URLs.

I'd recommend adding a new UUID field to your model, and remapping your views to use it, instead of the PK, for object lookups.

Answer 3

A really simple solution is simply encrypting the ID before sending it out to an external source. You can decrypt it on the way back in.

Answer 4

Keep the AUTO_INCREMENT, but pass it around in a semi-secret way: In a cookie. It takes a bit of coding to establish the cookie, set it, and read it. But cookies are hidden from all but serious hackers.