How to apply function row by row, and save csv at each step (append)?

Question

I'm using following script (Python 3) with the intention to

apply a function row by row and
write each resulting line into a csv.

More precisely, I just like to append each row to the existing file.

Code:

with open('file.csv', 'a') as f:  
    df['newcolumn'] = df.apply(some_function, axis=1, result_type='expand')
    df.to_csv(f, mode='a', header=f.tell()==0, encoding='utf-8', index=False)

The issue with this code: Apparently, the script does not write line by line into the *.csv, but only creates the full csv when all lines are computed. For any error in the script, this implies that it ends up with zero content.

How can I re-write the script so that the file.csv is updated in every step, and not empty until completion?

Thank you!

Why don't you try reading all the data into a dataframe. apply the function to all the rows using iterrows(). and then save the dataframe to csv. I think it will do the same — Parijat Bhatt, Aug 11 '19 at 22:50

adnanmuttaleb · Answer 1 · 2019-08-10T15:44:58.953

you can alter the code of your function to return some default or fallback value if something went wrong, by doing this you can save your dataframe as csv with one call.

def some_function(row):
    try:
        #return your results of processing 
    except:
        #return a default, fallback or None if error occurred

But this will save all your rows, if you do not want that you can filter your dataframe with some condition then you save it as csv. something like that:

df[df['newcolumn'] != <Some Value>]

How to apply function row by row, and save csv at each step (append)?

1 Answers1