1

We have been working on an integration test suite for our Flutter project. We are able to run it locally on the Android and iOS simulators, and on chrome for our web version of the app. Now, we would like to run our integration test suite as part of our CI pipeline on GitLab.

Because we don't want our integration tests to interact with a live Firebase project, we are using the Firebase Emulators. We have a bash script that spins up the emulators and populates the Firestore emulator with data and Auth emulator with testing accounts (by running a typescript/javascript script).

However, I cannot find any documentation on how to (or if is even possible to) run the emulators and populate them with test data on Firebase Test Lab.

Zooming out, is it even advisable to use Test Lab to run your integration test suites? I would love some advise from someone who has worked on DevOps for a professional flutter project!

I have looked at the following pages but couldn't find an answer to my question: https://firebase.google.com/docs/test-lab/android/instrumentation-test https://firebase.google.com/docs/test-lab/android/continuous Also it is difficult to find good answers through google or stack overflow because Test Lab calls their virtual devices emulators too, yielding results about those emulators instead of the Firebase Emulators..

Some of the Google / Stack overflow searches that I tried:

  • "firebase emulators" "test lab" site:stackoverflow.com
  • flutter "firestore" "test lab"
  • firebase test lab run custom script
Maarten
  • 13
  • 2

1 Answers1

0

is it even advisable to use Test Lab to run your integration test suites?

I'd say yes – it's one of the most mature device lab platforms. It has good docs and is quite widely used. You just have to be aware of its limitations – one of which you've just hit :)

I cannot find any documentation on how to (or if is even possible to) run the emulators and populate them with test data on Firebase Test Lab.

It's not possible – you can't run any script before or after your tests end. You don't have any kind of shell access that'd allow you to run the Firebase Emulators.

PS I think you could try hacking around to make it work – for example by running Firebase Emulators on e.g GitHub Actions, exposing them to the internet, and then running tests on Test Lab (with API URLs in your app changed to the Firebase Emulators). But I doubt it's worth it – why not just create a second firebase project to use as "staging/test" environment?

Bartek Pacia
  • 1,085
  • 3
  • 15
  • 37
  • Thanks for the advice! I got so caught up in getting everything to run locally first that I forgot about the option to use a second a live firebase project. However, how would you deal with scaling up when the dev team gets bigger and we want to run multiple pipelines at the same time? We can't really share the same firebase backend as we want to deploy the backend code of the current commit/branch. I guess making multiple Firebase projects dedicated to tests would work? Also, do live firebase projects have the option to wipe all firestore / storage data and accounts? – Maarten Feb 08 '23 at 22:31
  • I think you're trying to do too much with Firebase and you should roll your own "traditional style" backend. I recommend reading a series of articles about Firebass, its pros, cons, and limitations: https://leancode.co/blog/why-firestore-firebase-pros-cons-reasons-to-love-it-part-i :) – Bartek Pacia Feb 08 '23 at 22:52
  • 1
    Unfortunately the decision to use Firebase was made 1.5 years ago and it is hard to change now. But thanks for the blog series, I'll read it just to be more aware of Firebase's limitations. – Maarten Feb 09 '23 at 08:21