Export Collections to BigQuery

Question

I have tried to export collection to BigQuery using Export Collections to BigQuery extension.
But I am not able to export all collections.

I need to export all collection and subcollection to BiqQuery using Export Collections to BigQuery extensions

I am able to export only one collection but can't export all.

Could anyone please help me to export all collections and subcollections?

score 1 · Accepted Answer · answered Jan 29 '20 at 14:08

1

You can try to export the information to a bucket or a local file by running the command gcloud firestore export gs://mybucket/my/path as mentioned here. This will export all the collections with a single command.

I recommend you to export it to GCS, however, you can load the info to BigQuery by following one of these methods.

There most likely is an issue with the extension you are using since extensions are still on beta, but you can still open a Feature request so that they add more functionalities to the extension in the future and/or report any bug directly.

Hope you find this useful!

answered Jan 29 '20 at 14:08

rsalinas

1,507
8
9

I already tried export but if I change any firebase collection that value does not change on bigquery. – Murugan Jan 31 '20 at 04:14
I need to if I change any firebase collection that changes automatically update to BigQuey dataset – Murugan Jan 31 '20 at 04:26
Then what you want to do is synchronize them. Currently I do not think there is any solution which would do this automatically, but what you can try to do is create a [Cloud Function with a Firestore Trigger](https://firebase.google.com/docs/functions/firestore-events) that modifies your Bigquery table similarly to what is being done in [this article.](https://cloud.google.com/solutions/streaming-data-from-cloud-storage-into-bigquery-using-cloud-functions) – rsalinas Jan 31 '20 at 08:59

score 1 · Answer 2 · answered Jul 21 '22 at 20:19

1

You can install a second extension "stream collections to biquery" to have a second collection exported to BQ. Just go to the menu extensions and you did the first time

answered Jul 21 '22 at 20:19

crisitianrojasn

11
1

score 0 · Answer 3 · answered Aug 12 '22 at 14:09

One thing that was helpful for me was to go to the Cloud Functions section for my Firebase project, hop over to the GCP console side of things, and download a zip of the source code deployed for my ext-firestore-bigquery-export-fsexportbigquery function.

In there, I noticed that the value of process.env.COLLECTION_PATH ultimately gets passed through to functions.firestore.document() which is just constructing a namespaced collection path, where wildcards can be used at any level in the path...and therein lies the magic.

If you set your Collection Path to simply be a wildcard variable, such as "{AnyCollection}", then the function will capture events for any and all collections in your Firestore DB. Only one instance of the extension is needed.

Overall solution:

Set Collection Path to "{AnyCollection}"
Set Table ID to something indicating that it's multi collection, such as "AnyCollection"
It can be helpful to set Wildcard Column field with Parent Firestore Document IDs to true, although not required.
- You also could interrogate the name of the collection out of the document_name column in your BigQuery table, which includes paths such as "projects/my-firebase-project/databases/(default)/documents/my-firestore-collection/U5OeEv7jZCY8ja0g2gLr".

Export Collections to BigQuery

3 Answers3