Exporting data from firebase firestore to MeiliSearch using Google Cloud Function

Question

I was working on exporting data from Firebase Firestore to MeiliSearch (and I am new to this) through Cloud Function, but the problem was that it was collecting all of the data but in MeiliSearch it was exporting only a chunk, say 400 out of 25k entries, and it became less and less with exporting the data by triggering the function again and again.

The meiliSearch is on digital Ocean.

Note: It also cost me because Firebase says I had made 27 million calls to firestore.

const functions = require("firebase-functions");
const admin = require("firebase-admin");
const axios = require('axios');
const { default: MeiliSearch } = require("meilisearch");

exports.importToMeili = functions.https.onRequest(async (req, res) => {

    const config={
        host: "http://46.101.197.194",
        apiKey: 'My API Key'
    }

    admin.initializeApp();
    const db = admin.firestore();
    const userCollection = await db.collection("artikli").get();
    const client=new MeiliSearch(config);

    const index=0;
    for(index;index<24752;index++){
        await client.index('artikli').addDocuments([{
            objectID: userCollection.docs[index].id,
            naziv: userCollection.docs[index].n,
            marka: userCollection.docs[index].m,
            model: userCollection.docs[index].mo,
            katBr: userCollection.docs[index].kb,
            id: userCollection.docs[index].id,
            cijena: userCollection.docs[index].c,
            kolicina: userCollection.docs[index].ko,
            lokacija: userCollection.docs[index].l,
            opis: userCollection.docs[index].o,
            image: userCollection.docs[index].u,
            name: userCollection.docs[index].n,
            brand: userCollection.docs[index].brand,
            price: userCollection.docs[index].c.isEmpty ? 0 : instance.c,
            images: userCollection.docs[index].u,
        }]);
    }

    userCollection.docs.map(async doc => {
        const instance = doc.data();

        await client.index('artikli').addDocuments([{
            objectID: doc.id,
            naziv: instance.n,
            marka: instance.m,
            model: instance.mo,
            katBr: instance.kb,
            id: instance.id,
            cijena: instance.c,
            kolicina: instance.ko,
            lokacija: instance.l,
            opis: instance.o,
            image: instance.u,
            name: instance.n,
            brand: instance.brand,
            price: instance.c.isEmpty ? 0 : instance.c,
            images: instance.u,
        }]);

        return {
            objectID: doc.id,
      naziv: instance.n,
      marka: instance.m,
      model: instance.mo,
      katBr: instance.kb,
      id: instance.id,
      cijena: instance.c,
      kolicina: instance.ko,
      lokacija: instance.l,
      opis: instance.o,
      image: instance.u,
      name: instance.n,
      brand: instance.brand,
      price: instance.c.isEmpty ? 0 : instance.c,
      images: instance.u,
        }
    });

    await axios.post(
        `http://46.101.197.194/indexes/artikli/documents`, 
        users,
        { headers: { "X-Meili-API-Key": "MY API KEY"} }
    )
    res.status(200).send()  
});

Now, I know that it is taking the whole table data from firestore but I don't know why it is not going in meilisearch.

I think you should split this into 2 separate tasks: 1) export the firestore database into a file, e.g. in Firebase storage 2) import the file into MeiliSearch This way you only pay once for the Firestore reads — thomas gotzsche, Jan 07 '22 at 18:16
@thomasgotzsche I don't have issue with the payment, I just want to know that why isn't it exporting all the data to meilisearch. — Munib Shah, Jan 07 '22 at 18:40

score 1 · Answer 1 · answered Jan 10 '22 at 21:41

MeiliSearch writes data asynchronously in order to continue to receive documents while the previous write requests are being processed by MeiliSearch.

What can happen is that your documents are still being indexed or that there was an error during indexing.

You can view the asynchronous tasks list by using the API endpoint GET /indexes/artikli/updates. This way you can know when the indexing is finished or why it may have failed. The javascript SDK provides methods to check if an update is finished. Related methods from the SDK

It is also advised to write your documents in several batches (the bigger the better) in order to mutualize the calculation of data structures that could be entirely recalculated every time you send a document one by one.

Exporting data from firebase firestore to MeiliSearch using Google Cloud Function

1 Answers1