GAE/P: Using deferred tasks and transactions for a counter

Question

I have a counter in my app where I expect that 99% of the time there will not be contention issues in updating the counter with transactions.

To handle the 1% times when it is busy, I was thinking of updating the counter by using transactions within deferred tasks as follows:

def update_counter(my_key):
    deferred.defer(update_counter_transaction)

@ndb.transactional
def update_counter_transaction(my_key):
    x = my_key.get()
    x.n += 1
    x.put()

For the occasional instances when contention causes the transaction to fail, the task will be retried.

I'm familiar with sharded counters but this seems easier and suited to my situation.

Is there anything I am missing that might cause this solution to not work well?

Answer 1

A problem may exist with the automatic task retries which at least theoretically may happen for reasons other than transaction colissions for the intended counter increments. If such undesired retry successfully re-executes the counter increment code the counter value may be thrown off (will be higher than the expected value). Which might or might not be acceptable for your app, depending on the use of the counter.

Here's an example of undesired defered task invocation: GAE deferred task retried due to "instance unavailable" despite having already succeeded

The answer to that question seems inline with this note on regular task queue documentation (I saw no such note in the deferred task queues article, but I marked it as possible in my brain):

Note that task names do not provide an absolute guarantee of once-only semantics. In extremely rare cases, multiple calls to create a task of the same name may succeed, but in this event, only one of the tasks would be executed. It's also possible in exceptional cases for a task to run more than once.

From this perspective it might actually be better to keep the counter incrementing together with the rest of the related logical/transactional operations (if any) than to isolate it as a separate transaction on a task queue.