0

how can i upload a large excel file and dataset (larger than 10 mb) to big query ? Can anyone help ?

I tried to research a way to do it, but everything i found was a little bit complicated. I just started working with SQL and I don't have much experience.

Abdel Salam Majdi Abu Ghazaleh
  • 1
  • Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. – Community Dec 24 '22 at 18:18

1 Answers1

0

!!! WARNING prior to answer.

The below answer will incur costs. You can use --dry-run to estimate the number of bytes that are processed for bq and then combine it with Pricing calculator:

https://cloud.google.com/products/calculator/

What you are trying to achieve, is possible by loading a dataset from a google cloud storage bucket.

To make it easy for new users, there is a "Guide me" option from GCP on how to upload datasets larger than 10 mb.

GCP "Guide me" site

Don't forget to first estimate the costs.

An example of how this can be used using a Google Storage URI is referenced in this page:

  bq load \
    --source_format=CSV \
    mydataset.mytable \
    gs://mybucket/mydata.csv \
    ./myschema.json

Again, use pricing calculator to estimate costs.

To be able to create buckets on a specific project, you need specific IAM permissions, also referenced here

aamdevsecops
  • 104
  • 8