Firestore trigger timeouts occasionally

Question

I have a Cloud Firestore trigger that takes care of adjusting the balance of a user's wallet in my app.

exports.onCreateTransaction = functions.firestore
    .document('accounts/{accountId}/transactions/{transactionId}')
    .onCreate(async (snap, context) => {
        const { accountId, transactionId  } = context.params;

        const transaction = snap.data();

        // See the implementation of alreadyTriggered in the next code block
        const alreadyTriggered = await firestoreHelpers.triggers.alreadyTriggered(context);

        if (alreadyTriggered) {
            return null;
        }

        if (transaction.status === 'confirmed') {
            const accountRef = firestore
                .collection('accounts')
                .doc(accountId);

            const account = (await accountRef.get()).data();
            const balance = transaction.type === 'deposit' ? 
                    account.balance + transaction.amount : 
                    account.balance - transaction.amount;

            await accountRef.update({ balance });
        }

        return snap.ref.update({ id: transactionId });
    });

As a trigger may actually be called more than once, I added this alreadyTriggered helper function:

const alreadyTriggered = (event) => {
    return firestore.runTransaction(async transaction => {
        const { eventId } = event;
        const metaEventRef = firestore.doc(`metaEvents/${eventId}`);
        const metaEvent = await transaction.get(metaEventRef);

        if (metaEvent.exists) {
            console.error(`Already triggered function for event: ${eventId}`);
            return true;
        } else {
            await transaction.set(metaEventRef, event);
            return false;
        }
    })
};

Most of the time everything works as expected. However, today I got a timeout error which caused data inconsistency in the database.

Function execution took 60005 ms, finished with status: 'timeout'

What was the reason behind this timeout? And how do I make sure that it never happens again, so that my transaction amounts are successfully reflected in the account balance?

Answer 1

That statement about more-than-once execution was a beta limitation, as stated. Cloud Functions is out of beta now. The current guarantee is at-least-once execution by default. you only get multiple possible events if you enable retries in the Cloud console. This is something you should do if you want to make sure your events are processed reliably.

The reason for the timeout may never be certain. There could be any number of reasons. Perhaps there was a hiccup in the network, or a brief amount of downtime somewhere in the system. Retries are supposed to help you recover from these temporary situations by delivering the event potentially many times, so your function can succeed.