list_jobs function min_creation_time error

Question

Below is my code in Django frame (python 2.7) to list the jobs in Bigquery. I want to filter to just the ones in last two weeks but the min_creation_time in the list_jobs() function does not work and errors out for some reason. Please suggest

from __future__ import unicode_literals
from django.shortcuts import render
import thd_gbq_tools as bq
# Create your views here.
from django.http import HttpResponse
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
from oauth2client.client import GoogleCredentials
from google.cloud import bigquery
import uuid
import os
import logging
import time
import json
from datetime import datetime,timedelta
from django.template import loader
from django.shortcuts import render
import pandas as pd
from collections import OrderedDict
from datetime import date



def home(request):


    credentials = GoogleCredentials.get_application_default()
    # Construct the service object for interacting with the BigQuery API.
    bq_conn = build('bigquery', 'v2', credentials=credentials)



    job_query_dict = []    

    import warnings
    warnings.filterwarnings("ignore")
    ###Create the big query client
    client =bigquery.Client(project='analytics-supplychain-thd')




    ###List the jobs in the client
    jobs = client.list_jobs(all_users= True)  # API request


    for job in jobs:
        job_create_timestamp = datetime.strptime((str(job.created).replace('+','.')).split('.')[0],'%Y-%m-%d %H:%M:%S')
        job_ended_timestamp = datetime.strptime((str(job.ended).replace('+','.')).split('.')[0],'%Y-%m-%d %H:%M:%S')
        job_query_dict.append([job.job_id, job.user_email , job_create_timestamp,job_ended_timestamp, job.state])

    Table1 = sorted(job_query_dict,key=lambda x: (x[2]), reverse=True)

    return render(request, 'j2_response.html', {'Table1':Table1})

This is the code I am using to assign the parameter that indicates the last 10 minutes for min_creation_time:

from datetime import datetime,timedelta 
from datetime import date 

ten_mins_ago = datetime.utcnow() - timedelta(minutes=10)

what error message are you seeing? Also, why are you instantiating a `googleapiclient` builder if you are not using it? — Willian Fuks, Jun 06 '18 at 22:16
I actually don't see where you are setting the time parameters. i.e. https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/list — Graham Polley, Jun 07 '18 at 08:17
Hey @Willian and Graham thanks for the comments. Here is the line where I use the min_creation_time is used jobs = client.list_jobs(all_users= True,max_results= 10,min_creation_time=ten_mins_ago) — Will_smith12, Jun 07 '18 at 14:12
Here is the error I get list_jobs() got an unexpected keyword argument 'min_creation_time' — Will_smith12, Jun 07 '18 at 14:15
What object type are you adding to the parameter `min_creation_time`? Note that it has to contain `tzinfo` (timezone info). — komarkovich, Jun 08 '18 at 14:24
from datetime import datetime,timedelta from datetime import date ten_mins_ago = datetime.utcnow() - timedelta(minutes=10) I am passing ten_mins_ago as the value to it but I dont think it even recognizes the parameter. This is the value used in google documentation here https://google-cloud-python.readthedocs.io/en/latest/bigquery/usage.html — Will_smith12, Jun 08 '18 at 14:28

score 1 · Answer 1 · answered Jun 13 '18 at 09:26

1

When indicating ten_mins_ago = datetime.utcnow() - timedelta(minutes=10) you are specifying that you want the BigQuery jobs that have been run for the last 10 minutes.

You can try this code snippet to list the BigQuery jobs made in the last 2 weeks:

from google.cloud import bigquery
from datetime import datetime, timedelta
from pytz import timezone

client = bigquery.Client(project = '[YOUR_PROJECT]')
local_timezone = timezone('US/Eastern')
two_weeks_ago = datetime.utcnow() - timedelta(days = 14)
local_two_weeks = local_timezone.localize(two_weeks_ago)

for job in client.list_jobs(all_users = True, max_results = 10, min_creation_time = local_two_weeks):
    print(job.job_id, job.user_email)

If this snippet works for you, you can integrate it into your code. Should you get any errors, please state them so we can look further into the issue.

answered Jun 13 '18 at 09:26

Rodrigo C.

1,114
7
13

Thanks for the snippet but I keep getting the same error . list_jobs() got an unexpected keyword argument 'min_creation_time' May it has to do with the google.cloud version I am using. What do you think – Will_smith12 Jun 13 '18 at 15:42
Which version are you using right now? The current version for the `google-cloud-bigquery` library is 1.3.0, while for google-cloud itself is 0.33.1 – Rodrigo C. Jun 14 '18 at 11:18
google-cloud==0.33.1 google-cloud-bigquery==1.2.0 google-api-python-client==1.7.3 google-auth==1.5.0 – Will_smith12 Jun 14 '18 at 14:16
What I would advise here is to upgrade the Python packages you are using, by running the command `pip install [PACKAGE_NAME] --upgrade`. That way you will get the newest versions of the packages. Then you can try the code snippet again and check if it works now. – Rodrigo C. Jun 14 '18 at 15:42

list_jobs function min_creation_time error

1 Answers1