5

The local Firestore emulator for firebase seems fairly slow when dealing with bulk data.

I'm trying to simulate a production batch import of ~5,000,000 documents, but after a few thousand records the server is so overwhelmed it slows to a rate of around 5 records/second.

On the topic of Transactions and Batched Writes firebase says...

Note: For bulk data entry, use a server client library with parallelized individual writes. Batched writes perform better than serialized writes but not better than parallel writes. You should use a server client library for bulk data operations and not a mobile/web SDK.

That implies to me that I should simply not use the mobile/web SDK, and should make sure I use a server-side SDK/API for imports.

I'm already using the server client library for node.js.

  const batch = firestore.batch();

  for (const record of records) {
    batch.set(
      firestore.collection(collection).doc(record.key),
      record,
    );
  }
  await batch.commit();

...and this issue makes me think the local emulator (or firestore in general) can't keep up with a high load.

Frank van Puffelen
  • 565,676
  • 79
  • 828
  • 807
Will Haley
  • 703
  • 1
  • 9
  • 25
  • 1
    did you ever get to the bottom of this ? I am experiencing exactly this issue with the firestore emulator. I tested a few things and it comes down to this. Let me know if you found a solution ? – Janpan Jun 29 '21 at 17:15
  • 1
    I did @Janpan. See https://github.com/firebase/firebase-tools/issues/3477 The UI itself is slowing down the emulator for bulk imports in my case. – Will Haley Jun 29 '21 at 23:00

0 Answers0