GKE Pod download speed of BigQuery API is very slow

Question

I have an image that performs a query on BigQuery and then downloads the result (about 8GB) with the following code:

query_job = client.query(my_sql_query)
query_result = query_job.result()
data_table = query_result.to_arrow(progress_bar_type='tqdm')

When I run the code locally the download speed is OK (100 mbs, basically my computer actual bandwidth). The problem is that when I run It on GKE cluster the download is very slow (With tqdm I can see that it will be about 2 hours).

I checked the bandwidth of my pod with the python package speedtest-cli and it is near 500 mbs.

Does your pod struggle with the resource allocation? Are you sure to have enough CPU/Memory to run smoothly your pod? — guillaume blaquiere, Dec 07 '21 at 15:38
@guillaumeblaquiere I made tests with high resources to check if there was the bottleneck and the problem persisted. I used 15 CPU and 40 Gi of RAM — Fran Arenas, Dec 08 '21 at 16:07
Can you run `curl -s -o /dev/null -w "%{time_total}\n" googleapis.com` and share the results ? — Wojtek_B, Dec 09 '21 at 14:37
@Wojtek_B The results of the command are between 0.009 and 0.003 — Fran Arenas, Dec 10 '21 at 07:43
At this point we can rule out networking issues. Maybe it's your storage speed that limits the performance ? Where do you store the data that you download from BigQuery ? — Wojtek_B, Dec 10 '21 at 11:06

score 3 · Accepted Answer · answered Dec 10 '21 at 10:53

3

The problem was that I wasn't installed the package google-cloud-bigquery-storage inside the pod so I was using the google-cloud-bigquery package to perform the download.

answered Dec 10 '21 at 10:53

Fran Arenas

103
1
6

GKE Pod download speed of BigQuery API is very slow

1 Answers1