We are working on some optimizations to decrease cold start time in our firebase cloud functions. As a result of this we'd like to compare the execution times before and after these changes were implemented.
Is there a reliable way to measure cold start time for a cloud function in the google cloud metrics explorer?
The closest metric I could find is execution time, but its unclear if that includes the time it takes to load the dependencies before running a function.
I dont think what you found includes the cold-start time. I'd think it's more accurate to calculate the difference between execution start and execution finish in an empty https-triggered cloud function. This way the difference would strictly be the cold start time.
I'm not aware of a way to calculate this systematically. This is because the cold-start has nothing to do with the function itself. It's just an initialisation operation in the server where the function is hosted. Meaning that for all intents and purposes, the function doesn't even exist before initialisation and thus can not have access to the timestamp at the start of the cold-start.
I guess you'll have to do this manually to somehow get an average.
Edit0:
Perhaps you can create a script that stores the current time right before calling the https function. And then stores another timestamp right after the function returns with a status of 200. Then you just calculate the difference between the 2 timestamps. You get a relatively pure cold-start time.
Edit1:
If your question only refers to cloud functions, you should consider bypassing cold start altogether (which is what we've been doing). We did it by scheduling a Cron job on Cloud Scheduler. Here's the https function that keeps our login token generator "warm". It simulates a user requesting a login token every 30 seconds but it deletes the token immediately. You could do something similar.
It's very important to return status 200 in Cloud Scheduler (no matter what happens). We don't want the function to return any error code because this will terminate the Cron job.
exports.PreventColdStart = functions.https.onRequest((req,res) => {
admin.firestore().collection(...).doc(...).set({...}).then(()=>{
function sleep (time) {
return new Promise((resolve) => setTimeout(resolve, time));
}
// Usage!
sleep(500).then(() => {
admin.firestore().collection(...).doc(...).delete().then(() => {
res.status(200).send('Successfully deleted');
}).catch((error) => {
res.status(200).send('Did not delete');
});
});
}).catch(()=>{
res.status(200).send('Could not find document');
})
})
You can check all the Firebase Database Performance usage with several options to choose to measure this and an overview of what each one offers. But what will focus on is the last part in the Cloud monitoring paragraph.
The steps for setting up Cloud Monitoring are described in Monitor Database Usage.
If you keep reading this page there is a Cloud Monitoring overview of the metrics you can use and this can help you to navigate through a list you can find in the next page that is the Monitor Database Usage. This page also includes more details of the mentioned options you have to measure the performance of firebase and finally, in the Cloud Monitoring section there is a link to a list of all the metrics you can use, looking through it I think the closest to your specific case it is the instance/cpu/scheduler_wait_time by its description:
Wait time is the time a vCPU is ready to run, but unexpectedly not scheduled to run. The wait time returned here is the accumulated value for all vCPUs. The time interval for which the value was measured is returned by Monitoring in whole seconds as start_time and end_time. This metric is only available for VMs that belong to the e2 family or to overcommitted VMs on sole-tenant nodes. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. instance_name: The name of the VM instance.
Feel free to navigate through them and choose the one that covers your needs, I found other metrics that can also be helpful in your case.
