Create URLs for different data frames

Question

I have a data frame that I split into different data frames of size 100 (to be able to make Python able to process it). Therefore, I get different data frames (df1 to df..). For all those data frames, I want to create an URL as shown below.

When I use type(df), it shows me it is a data frame, however, when I use for j in dfs: print(type(j)), it is shown it is a string. I need the data frame to make it able to create the URL.

Can you please help me what the loop for creating the urls for all data frames could look like?

Thank you so much for your help!

df = pd.DataFrame.from_dict(pd.json_normalize(tweets_data), orient='columns')


n = 100  #chunk row size
list_df = [df[i:i+n] for i in range(0,df.shape[0],n)]

dfs = {}
for idx,df in enumerate(list_df, 1):
    dfs[f'df{idx}'] = df

type(df1)

for j in dfs:
    print(type(j))

def create_url():
     url = "https://api.twitter.com/2/tweets?{}&{}".format("ids=" + (str(str((df1['id'].tolist()))[1:-1])).replace(" ", ""), tweet_fields)
     return url

df Out[144]: created_at ... coordinates.coordinates 12400 Wed Aug 04 21:57:59 +0000 2021 ... NaN 12401 Wed Aug 04 21:58:07 +0000 2021 ... NaN — Data_Science_110, Aug 20 '21 at 11:14
It is basically a data frame of all the Twitter data, e.g., creation data, coordinates, text, but I am only interested in the column id — Data_Science_110, Aug 20 '21 at 11:15
My final string looks similar to that: 'https://api.twitter.com/2/tweets?ids=1422935798802960386,1422935813466243072, ... — Data_Science_110, Aug 20 '21 at 11:16
if `dfs` is dictionary then `for j in dfs:` gives you only keys - which are string. You need `for j in dfs.values():` or `for key, j in df.items():` — furas, Aug 20 '21 at 11:42

furas · Answer 1 · 2021-08-20T13:20:50.247

0

dfs is dictionary so for j in dfs: gives you only keys - which are string.

You need .values()

for j in dfs.values():

or .items()

for key, j in df.items():

or you have to use dfs[j]

for j in dfs:
    print(type( dfs[j] ))

EDIT:

Frankly, you could do it all in one loop without list_df


import pandas as pd

#df = pd.DataFrame.from_dict(pd.json_normalize(tweets_data), orient='columns')
df = pd.DataFrame({'id': range(1000)})

tweet_fields = 'something'

n = 100  #chunk row size

for i in range(0, df.shape[0], n):
    ids = df[i:i+n]['id'].tolist()
    ids_str = ','.join(str(x) for x in ids)
    url = "https://api.twitter.com/2/tweets?ids={}&{}".format(ids_str, tweet_fields)
    print(url)

You can also use groupby and index if index uses numbers 0,1,...

for i, group in df.groupby(df.index//100):
    ids = group['id'].tolist()
    ids_str = ','.join(str(x) for x in ids)
    url = "https://api.twitter.com/2/tweets?ids={}&{}".format(ids_str, tweet_fields)
    print(url)

edited Aug 20 '21 at 13:20

answered Aug 20 '21 at 11:45

furas

134,197
12
106
148

Thank you so much, that helps! :) However, now I get the right format but I am not able to create a loop which creates the URL for all id's within one data frame, do you maybe know how I can do it? – Data_Science_110 Aug 20 '21 at 12:38
you don't need another loop but you have to run `create_url` inside current loop and you should create function with argument `def create_url(df)`. Better create minimal working code with example data in code - so we could simply copy it and run it. – furas Aug 20 '21 at 13:04
frankly, you could do all in single `for`-loop - `for i in range(0,df.shape[0],n)` - without creating `list_df` – furas Aug 20 '21 at 13:08

Create URLs for different data frames

1 Answers1