3

I searched and tested different ways to find if I can be able to split bigquery dataframe into chunks of 75 rows, but couldn't find a way to do so. here is the senario:

I got a very large bigquery dataframe (millions of rows) using python and gcp SDK:

    from google.cloud import bigquery

    def returnBQResult(sqlQuery):
       bqClient = bigquery.Client()
       query_job = bqClient.query(sqlQuery)
       return query_job.to_dataframe()

I am looking for something similar to below numpy code but using dask and doing it parallel:

       dfBQResults=returnBQResult(sql)
       result=numpy.array_split(dfBQResults,75)
MT467
  • 668
  • 2
  • 15
  • 31

0 Answers0