I searched and tested different ways to find if I can be able to split bigquery dataframe into chunks of 75 rows, but couldn't find a way to do so. here is the senario:
I got a very large bigquery dataframe (millions of rows) using python and gcp SDK:
from google.cloud import bigquery
def returnBQResult(sqlQuery):
bqClient = bigquery.Client()
query_job = bqClient.query(sqlQuery)
return query_job.to_dataframe()
I am looking for something similar to below numpy code but using dask and doing it parallel:
dfBQResults=returnBQResult(sql)
result=numpy.array_split(dfBQResults,75)