How to import test data into Firebase Firestore emulator from production for easy testing?

Question

I am building a firebase function that listens to a trigger and sends a push notification to users. The trigger is based on firestore data that's too complex for me to manually re-recreate every time on firestore emulator.

I tried emulating functions, but hooking up to firestore production, but seems like in this case, trigger functions don't work

What I want to do is export my data from my production Firestore then import it into my emulator firestore, so that at least, the copy I am working with is closely mimicking what I have on prod; to be clear, the data is small, so I am not worried about downloading terabytes of data.

I found a way of importing data into an emulator, but not sure if this would work with production data or how I would dump the data from production.

firebase emulators:start --import=./some-directory

Answer 1

You can manually export from Production and then import into the Firestore emulator.

Start by doing an export of your production data to a file. In the example, I export just the "users" collection within an onRequest function so I can kick-off by calling the URL. Note, this this function will not handle subcollections.

exports.exportFirestore = functions.https.onRequest(async (req, res) => {
  const fs = require("fs");

  const collection = "users";
  const fileName = "fs-export.json";
  const exportedData = {};
  exportedData[collection] = {};

  await admin
    .firestore()
    .collection(collection)
    .get()
    .then((snapshot) => {
      return snapshot.forEach((doc) => {
        exportedData[collection][doc.id] = doc.data();
      });
    })
    .catch(console.error);

  fs.writeFile(fileName, JSON.stringify(exportedData), (err) => {
    if (err) {
      console.log(err);
    } else {
      console.log("Firestore Export Complete.");
      res.send("Firestore Export Complete.");
    }
  });
});

Next, import the file into the local Firestore emulator. Very important: be sure to have the Firestore emulator running otherwise you'll be importing your data back to prod. Be sure to check that Firestore is running by checking the web UI: http://localhost:4000. Start the emulator with: firebase emulators:start

exports.importFirestore = functions.https.onRequest(async (req, res) => {
  const fs = require("fs");

  let collection;
  const fileName = "fs-export.json";
  const exportedData = {};
  exportedData[collection] = {};

  fs.readFile(fileName, "utf8", async (err, data) => {
    if (err) {
      return console.log(err);
    }

    const arr = JSON.parse(data);

    const batch = admin.firestore().batch();

    for (let i in arr) {
      collection = i;
      for (let doc in arr[i]) {
        if (arr[i].hasOwnProperty(doc)) {
          const ref = admin.firestore().collection(collection).doc(doc);
          batch.set(ref, arr[i][doc]);
        } else {
          console.log("Missing:", JSON.stringify(doc, null, 2));
        }
      }
    }

    await batch
      .commit()
      .then(() => {
        return console.log("Import to Firestore Complete");
      })
      .catch(console.error);

    return res.send("Import to Firestore Complete");
  });
});

Answer 2

One way to persist your data is by exporting to directory on exit and on start import the same directory

firebase emulators:start  --import=./mydir --export-on-exit mydir 

Answer 3

You can do this using gcloud (Google Cloud's CLI). It does take several commands:

First, create a firestore export of your Firestore staging (Using production database for local development could be potentially dangerous) database (Make sure this bucket exists first!):

gcloud firestore export gs://{BUCKET}/{DIRECTORY}

Then, download the data from this bucket:

gsutil -m cp -r gs://{BUCKET}/{DIRECTORY} ./{LOCAL_DIRECTORY_NAME}

Finally, run the emulators with this command:

firebase emulators:start --import ./{LOCAL_DIRECTORY_NAME}

This should give you a local copy of your staging database to work on.

Answer 4

My answer is heavily inspired by @Geoffrey Bourne's answer, but I had to modify some stuff and figure out more details to get it working.

First, I upload the exportFirestore to Cloud Functions (production). When I run it through this URL https://us-central1-<project-id>.cloudfunctions.net/exportFirestore, I get a file DOWNLOADED, as Cloud Functions are read-only

The below code is for one collection named fl_content, I'll consider expanding it to multiple collections

export const exportFirestore = functions.https.onRequest(async (req, res) => {
    const collection = "fl_content";
    const exportedData: any = {};
    exportedData[collection] = {};

    await admin
        .firestore()
        .collection(collection)
        .get()
        .then((snapshot) => snapshot.forEach((doc) => exportedData[collection][doc.id] = doc.data()))
        .catch(console.error);

    const data = JSON.stringify(exportedData);
    res.setHeader('Content-disposition', 'attachment; filename=fire-export.json');
    res.setHeader('Content-type', 'application/json');
    res.write(data, function () {
        res.end();
    });
})

Once you have the file fire-export.json downloaded, put it inside the functions folder. Then open the URL for the import function (locally) http://localhost:5001/<project-id>/us-central1/importFirestore. Make sure the collection variable is the same in the export and import.

export const importFirestore = functions.https.onRequest(async (req, res) => {
    const fs = require("fs");
    const collection = "fl_content";
    const fileName = "fire-export.json";
    const exportedData: any = {};
    exportedData[collection] = {};

    fs.readFile(fileName, "utf8", async (err: any, data: any) => {
        if (err) {
            res.send(err);
            functions.logger.error(err)
            return;
        }
        const arr = JSON.parse(data);
        const batch = admin.firestore().batch();

        for (const i in arr) {
            for (const doc in arr[i]) {
                if (arr[i].hasOwnProperty(doc)) {
                    const ref = admin.firestore().collection(collection).doc(doc);
                    batch.set(ref, arr[i][doc]);
                } else {
                    functions.logger.error("Missing:", JSON.stringify(doc, null, 2));
                }
            }
        }

        await batch
            .commit()
            .then(() => console.log("Import to Firestore Complete"))
            .catch(console.error);

        res.send("Import to Firestore Complete");
    });
});