How to improve preformance of firestore cache query

Question

I am developing a PWA, which displays a list of transactions (transaction is an object with ~10 fields). I am using firestore for storage and realtime updates and I have also enabled persistance.

I want my application to have all the data in memory and I want to take care of displaying only necessary information myself (e.g. using virtual scrolling for transaction list). Due to this reason I listen to the whole collection (a.k.a the transactions).

At the start of the app, I want to make sure the data is loaded so I use one time cache query to get the transactions. I would expect the query to be nearly instantaneous, but on laptop it takes around ~1 second to get the initial data (and I also have another collection which I fetch from cache and this resolves after ~2 seconds after transactions request). For mobile it takes around ~9seconds (loading on mobile, loading on laptop)

I want my app to feel instantaneous, but I takes a few seconds until the data is in place. Note, that I am not doing any advanced queries (I just want to load the data to memory).

Am I doing something wrong? I have read Firestore docs, but I don't think the amount of data that I have in cache should cause such bad performance.

UPDATE: Even if I limit the initial query to just load 20 documents. It still takes around ~2 seconds to retrieve them.
UPDATE 2: The code looks like this:

export const initializeFirestore = (): Thunk => (dispatch) => {
  const initialQueries: Array<Promise<unknown>> = []
  getQueries().forEach((query) => {
    const q = query.createFirestoneQuery()
    initialQueries.push(
      q
        .get({
          source: 'cache',
        })
        .then((snapshot) =>
          dispatch(firestoneChangeAction(query, snapshot, true)),
        ),
    )
    q.onSnapshot((change) => {
      dispatch(firestoneChangeAction(query, change))
    })
  })

  console.log('Now I am just waiting for initial data...')
  return Promise.all(initialQueries)
}

Answer 1

You may be interested by the smart approach presented by Firebase engineers during the "Faster web apps with Firebase" Session of the Firebase Summit 2019 (You can watch the video here: https://www.youtube.com/watch?v=DHbVyRLkX4c).

In a nutshell, their idea is to use the Firestore REST API to make the first query to the database (which does not need to download any SDK), and in parallel, dynamically import the Web SDK in order to use it for the subsequent queries.

The github repository is here: https://github.com/hsubox76/fireconf-demo

---

I paste below the content of the key js file (https://github.com/hsubox76/fireconf-demo/blob/master/src/dynamic.js) for further reference.

import { firebaseConfigDynamic as firebaseConfig } from "./shared/firebase-config";
import { renderPage, logPerformance } from "./shared/helpers";

let firstLoad = false;

// Firestore REST URL for "current" collection.
const COLLECTION_URL =
  `https://firestore.googleapis.com/v1/projects/exchange-rates-adcf6/` +
  `databases/(default)/documents/current`;


// STEPS
// 1) Fetch REST data
// 2) Render data
// 3) Dynamically import Firebase components
// 4) Subscribe to Firestore



// HTTP GET from Firestore REST endpoint.
fetch(COLLECTION_URL)
  .then(res => res.json())
  .then(json => {
    // Format JSON data into a tabular format.
    const stocks = formatJSONStocks(json);

    // Measure time between navigation start and now (first data loaded)
    performance && performance.measure("initialDataLoadTime");

    // Render using initial REST data.
    renderPage({
      title: "Dynamic Loading (no Firebase loaded)",
      tableData: stocks
    });

    // Import Firebase library.
    dynamicFirebaseImport().then(firebase => {
      firebase.initializeApp(firebaseConfig);
      firebase.performance(); // Use Firebase Performance - 1 line
      subscribeToFirestore(firebase);
    });
  });

/**
 * FUNCTIONS
 */

// Dynamically imports firebase/app, firebase/firestore, and firebase/performance.
function dynamicFirebaseImport() {
  const appImport = import(
    /* webpackChunkName: "firebase-app-dynamic" */
    "firebase/app"
  );
  const firestoreImport = import(
    /* webpackChunkName: "firebase-firestore-dynamic" */
    "firebase/firestore"
  );
  const performanceImport = import(
    /* webpackChunkName: "firebase-performance-dynamic" */
    "firebase/performance"
  );
  return Promise.all([appImport, firestoreImport, performanceImport]).then(
    ([dynamicFirebase]) => {
      return dynamicFirebase;
    }
  );
}

// Subscribe to "current" collection with `onSnapshot()`.
function subscribeToFirestore(firebase) {
  firebase
    .firestore()
    .collection(`current`)
    .onSnapshot(snap => {
      if (!firstLoad) {
        // Measure time between navigation start and now (first data loaded)
        performance && performance.measure("realtimeDataLoadTime");
        // Log to console for internal development
        logPerformance();
        firstLoad = true;
      }
      const stocks = formatSDKStocks(snap);
      renderPage({
        title: "Dynamic Loading (Firebase now loaded)",
        tableData: stocks
      });
    });
}

// Format stock data in JSON format (returned from REST endpoint)
function formatJSONStocks(json) {
  const stocks = [];
  json.documents.forEach(doc => {
    const pathParts = doc.name.split("/");
    const symbol = pathParts[pathParts.length - 1];
    stocks.push({
      symbol,
      value: doc.fields.closeValue.doubleValue || 0,
      delta: doc.fields.delta.doubleValue || 0,
      timestamp: parseInt(doc.fields.timestamp.integerValue)
    });
  });
  return stocks;
}

// Format stock data in Firestore format (returned from `onSnapshot()`)
function formatSDKStocks(snap) {
  const stocks = [];
  snap.forEach(docSnap => {
    if (!docSnap.data()) return;
    const symbol = docSnap.id;
    const value = docSnap.data().closeValue;
    stocks.push({
      symbol,
      value,
      delta: docSnap.data().delta,
      timestamp: docSnap.data().timestamp
    });
  });
  return stocks;
}

Answer 2

You're not doing anything wrong. The query will take as much time as it needs to finish. This is why many sites use a loading indicator.

For the first query in your app, it's going to include the time it takes to fully initialize the SDK, which might involve asynchronous work beyond more than just the query itself. Also bear in mind that reading and sorting data from local disk isn't necessarily "fast", and that for larger amounts of documents, the local disk cache read might even be more expensive than the time it would take the fetch the same documents over the network.

Since we don't have any indication of how many documents you have, and how much total data you're trying to transfer, and the code you're using for this, all we can do is guess. But there's really not much you can do to speed up the initial query, other than perhaps limiting the size of the result set.

If you think that what you're experiencing is a bug, then please file a bug report on GitHub.