Generate schema views using 'Export Collections to BigQuery' extension : fs-bq-schema-views

Question

https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/GENERATE_SCHEMA_VIEWS.md

I have tried to create schema-views script but not able to create.

could anyone please help me with how to create step1?

Answer 1

Refer this doc for understanding BigQuery Schemas - Specifying a schema. Have explained more below.

Creating Schema for Firestore Collection Document -

• Schema is simply a json object which has a fields array.

{
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "age",
      "type": "number"
    }
  ]
}

• In the fields array, each item will have a name and type property. name is the fieldName you have used in your collections document and type is the dataType.
• For example - My Collection document has below interface (truncated):

So my schema is as below (note - have not added everything for brevity):

{
  "fields": [{
      "name": "awb",
      "type": "string"
    },
    {
      "name": "rfn",
      "type": "string"
    },
    {
      "name": "customerId",
      "type": "string"
    },
    {
      "name": "shipmentStatus",
      "type": "string"
    },
    {
      "name": "amount",
      "type": "number"
    }
  ]
}

Now, you need to simply create your schema in any directory in your machine and then run the npx @firebaseextensions/fs-bq-schema-views command from that directory.

This will open an interactive cli section where in you can add your projectId, datasetId etc, as below:

After that the schema will be created for you and you can check the same in BigQuery UI.

NOTE -

• Please ensure you have gcloud installed and configured. Steps are again easy, please follow Installing Cloud SDK
• Before running the npx command, do not forget to run gcloud auth application-default login

TIP -

• As a good practice, you may create a directory at the root of your project and store all schemas there. This will help to keep track of all the changes if you are using any version control system like git (provided you checkin)

This is how have done -

Answer 2

I had the same bug and it turned out, that I structured my .json file wrong.

The accurate structure is as follows:
In the schema json file the root of the configuration must have a fields array that contains objects which describe the elements in the schema, every object in the fields array will have a name and a type property. name is the fieldName you have used in your collections document in firestore and type is the dataType. If one of the objects is of type map, it must specify its own fields array describing the members of that map (Again exactly same as the root array of the configuration).

• for more details on how to structure the json schema file, kindly refer to the documentation.

    {
        "fields": [
            {
                "name": "uid",
                "type": "string"
            },
            {
                "name": "age",
                "type": "number"
            }
        ]
    }