1

I've (seemingly) successfully used pagination to iterate through Airtable records through a Python API call. But I'm having trouble understanding how to append the data from the first call and all of the iterations get calls. My code is below:

while offset is not None:
    next_url = url +'&offset='+ offset
    next_response = requests.get(next_url, params=params, headers=headers)
    output = next_response.json()
    for record in output["records"]:
       print(json.dumps(record))
    if "offset" in output:
        offset = output["offset"]
    else:
        offset = None 

        x = pd.DataFrame(output)
        x = x['records'].apply(pd.Series)
        x = x['fields'].apply(pd.Series)
        print(x)

        Series = {'offset': 'itrSuqdXIeBNUIR9R/rec0Nx6MPArR8mew8',
         'records': [{'createdTime': str,
                      'fields': {'*CHANNEL': str,
                                 '*START/LAUNCH/LOTS': str,
                                 '*STATUS': str,
                                 'CREATIVE OPS': str
                                 }
                                }]
                            }

I'm then trying to create a data frame of the appended dataset. Where in the loop do I append the data?

betontalpfa
  • 3,454
  • 1
  • 33
  • 65
Jordan Olson
  • 31
  • 2
  • 1
    By the way, the IDs in offset are not keys or passwords. They are just the offset IDs. – Jordan Olson Nov 21 '19 at 18:26
  • 1
    This post helped me figure out the append issue: https://stackoverflow.com/questions/52768509/how-to-append-several-dictionaries-while-looping-through-pagination-api-pyth . – Jordan Olson Nov 21 '19 at 19:55

1 Answers1

2

I figured it out:

while offset is not None:
next_url = url +'&offset='+ offset

next_response = requests.get(next_url, params=params, headers=headers)

output = next_response.json()

for record in output["records"]:
   print(json.dumps(record))
   all_pages.append(record)
if "offset" in output:
    offset = output["offset"]
else:
    offset = None 

    x = pd.DataFrame(all_pages)
    # x = x['records'].apply(pd.Series)
    x = x['fields'].apply(pd.Series)
    print(x)

I no longer needed to apply the record attribute to the dataset. I just needed to ask for the fields. Now I have all records in the view for specified fields in all one data frame!

Jordan Olson
  • 31
  • 2