What should I use as the document key to maintain idempotency?

Question

What should I use as the document key to maintain idempotency?

I'm building a text messaging application that uses CouchDB (with PouchDB on the client) to store messages locally. Twilio (SMS provider) generates an ID for each message, and I use that as the CouchDB document ID. This way fetching messages from Twilio's API is idempotent -- if I come across the same message twice, it will only store one copy in my database.

// twilio API /messages
[
  {smsid: 123, body: 'foo'},
  {smsid: 456, body: 'bar'}
]

// transformed into couchdb docs
[
  {id: 123, doc: {_id: 123, body: 'foo'}},
  {id: 456, doc: {_id: 456, body: 'bar'}}
]

This is easy to do when fetching messages from twilio. But when the user sends an outbound message from the client application, there is no twilio ID yet because it hasn't been sent to twilio yet.

A traditional approach would involve POSTing the message to some endpoint on my server, and have the server send it to twilio, then add the record to the database once it has the smsid from twilio's response. The problem with this is (a) there's a noticeable delay from when the user presses "send" and when the message shows up in the UI, and (b) we can't take advantage of couchdb's auth system.

Instead, I have it setup so the client generates a random ID, and inserts it into the database (via pouchdb w/sync). The server then watches for new outbound records added and dispatches them to twilio.

This approach works fine, but if I GET /messages again, it's no longer idempotent -- it would create an additional record for the outbound message because I don't have a couchdb document with that message's smsid as its key (it didn't have an smsid when it was added to couchdb).

Is there a way around this or a better approach?

Answer 1

An idea to make this work is that you must rely on other data from each message, and ignore Twilio's smsid.

Perhaps hashing together the user id, the message body and an abrangent version of the timestamp (for example, int(UNIX-TIMESTAMP-IN-SECONDS/100) will tolerate a delay of 100 seconds between the time your server gets the message and Twilio acknowledges it).

Answer 2

Thanks for your replies. This was a tough one. @rnewson from #couchdb in freenode was kind enough to spend some time thinking about this one and proposed a solution that worked out great:

• Message documents in couchdb use an arbitrary _id that can be generated by the server or the client
• When the client sends a message, it generates an arbitrary _id and puts it into the database. The server observes this and dispatches it to twilio, then updates the database document by adding a twilio_id property to the document
• I created a view to index the documents by twilio_id
• When the server starts, it fetches the latest messages from twilio. In order to prevent adding duplicate records to the database, it queries the above view for each twilio id. For each match, it uses the match's _id and _rev to perform an update. For records with no matches, it generates a new arbitrary _id to perform an insert.

For anyone curious, here's the code.

Thanks again for your responses!