Short unique IDs

Question

I'm designing a HTTP-service, with capacity of up to 500 million requests per day (served by more than one independent machine).

For each request I have to generate unique ID and return it to user. ID must be 100% unique within a window of 10 minutes. (1 day is preferred, globally unique IDs are ideal.) No server-server communication must be needed to generate that ID.

Silly pseudo-session example:

Client: GET /foo

Server: Content-Type: text/xml

        <root>
            <id>ab9d1972-2844-11e0-86b2-000c29544403</id>
            <other_data/>
        </root>

In previous generation of this HTTP service I used UUIDs.

I'm happy with UUIDs, but there is one problem: they are too long. On that number of requests, this extra size in noticeable in disk space waste for log files.

What is the best way to create a short, but unique identifier? To make things worthwhile, I guess, algorithm should produce at most half of UUID length while being unique for all day long (10 minutes should be even shorter).

Ideally, suggested algorithm would have sane, lightweight production-quality implementation in plain C.

Update: Generated ID should not require URI-encoding when passed in the GET request.

Lazy question (sorry, it is too late at night to do math): how long is UUID if encoded with ascii85 from binary? — Alexander Gladysh, Jan 29 '11 at 00:35
@Alexander: Number of digits is `ceil(log(max_val)/log(num_different_chars))`. — Oliver Charlesworth, Jan 29 '11 at 00:40
ASCII85 encodes 4 bytes in 5 characters. However, it is not *really* URI or human-friendly. (UUID is 128bits is 16 bytes is 20 characters ASCII85). — , Jan 29 '11 at 00:43
As far are making it unique, it depends upon exact requirements, but consider an approach like [twitter snowflake (twitter message numbers)](http://engineering.twitter.com/2010/06/announcing-snowflake.html) -- it uses only 64bits but a careful selection of machine/worker identification, time, and counters to guarantee uniqueness within the environment. Much more "guessable", but that's a weak reason/concern not to use a more problem-space refined approach. — , Jan 29 '11 at 00:46
@pst: why is ASCII85 not URI-friendly? (human-friendliness is not an issue) 20 characters is nice! — Alexander Gladysh, Jan 29 '11 at 00:50
@Alexander Gladysha While Base64 has one (or two?) characters that must be escaped in a URI, ASCII85 contains far more. URI encoding != URI friendly, and it a real bummer to look at in a location bar. — , Jan 29 '11 at 00:53
@pst: ah! you're right... no, it would not do then. I need something that would not require URI encoding. Tripling the length is not good. — Alexander Gladysh, Jan 29 '11 at 00:54

score 5 · Accepted Answer · answered Jan 29 '11 at 00:21

5

Give each machine a unique prefix. Give each machine a counter. To generate an ID, increment the counter, and append its value to the prefix.

If you want to obfuscate the IDs, encrypt them - a cipher is a reversible transformation, so applying it to unique values will produce unique values.

answered Jan 29 '11 at 00:21

Tom Anderson

46,189
17
92
133

2

Perhaps also make each ID three parts: machineid-counter-randomkey to eliminate ID prediction attacks. – Lawrence Dol Jan 29 '11 at 00:25
Good idea. Can you suggest a really fast cipher? – Alexander Gladysh Jan 29 '11 at 00:25
Also: How short do you think the ID can be if it is generated in your way? – Alexander Gladysh Jan 29 '11 at 00:27
@Alexander: 10 minutes at 500 million hits per day is 3.7 million. 7 digits ought to be enough for the counter (use 8 or 9 for headroom!). – Oliver Charlesworth Jan 29 '11 at 00:29
@Alexander: Note, that's 6 digits in hex. – Oliver Charlesworth Jan 29 '11 at 00:36
@Oli: or even less in ascii85! :-) – Alexander Gladysh Jan 29 '11 at 00:37
@Software Monkey: How long should be the randomkey? – Alexander Gladysh Jan 29 '11 at 00:38
@Alexander: It's a function of how difficult you want to make life for a hacker determined to generate a valid ID. You'd need to base it on the lifetime of an ID, and the number of attacks per second that a hacker could generate. – Oliver Charlesworth Jan 29 '11 at 00:45
Assuming alpha-numeric mixed case, dropping vowels to avoid objectionable words, therefore B-Z,0-9,~AEIOU, that gives you base 52 character set. So 10 characters would be 52^10=144,555,105,949,057,024 combinations; or 5 characters would be 380,204,032. Somewhere between 5 and 10 characters for the random part. – Lawrence Dol Jan 29 '11 at 02:15
@Alexander: AES/ECB with a 128-bit key has been benchmarked at 15-20 cycles per byte, which for a single 16-byte block means up to 320 cycles. That's peanuts compared to the work you'll have to do on IO (or even XML handling!) for each message. There are ciphers which may be even faster (eg XTEA), but i can't find solid benchmarks for them. – Tom Anderson Jan 30 '11 at 11:28
@Alexander: if you want a fast cipher, choose AES1 bit :) – JqueryToAddNumbers Apr 14 '12 at 05:19

score 2 · Answer 2 · answered Jan 29 '11 at 00:20

2

A few thoughts:

500 million requests a day. Really?
Use UUIDs.
If required, don't use HTTP (as that's the more significant overhead) and transfer the UUID in a binary form.
You need a certain amount of bytes to guarantee that your server returns a truly unique ID.
How about using UDP?

Anyway, what the heck are you trying to do?

answered Jan 29 '11 at 00:20

BastiBen

19,679
11
56
86

500M, really (that is target top capacity, estimated actual load is more like 100M). HTTP and TCP/IP is a must, unfortunately. – Alexander Gladysh Jan 29 '11 at 00:24
also, 500M/day should be within c10k limits, what is so surprising about it? – Alexander Gladysh Jan 29 '11 at 00:29

Short unique IDs

2 Answers2