2

my first post here in stackoverflow and trying to dip my feet into python by writing a program that calls data from an API of an online game I play :)

I've written the below code to create a .csv file if it doesn't exist, and then use a for loop to call an API twice (each with different match IDs). The response is in JSON, and the idea is that if the file is empty (i.e. newly created), it will execute the if statement to write headers in, and if it's not empty (i.e. the headers have already been written in), then to write the values.

My code returns a .csv with the headers written twice - so for some reason within the for loop the file size doesn't change even though the headers have been written. Is there something i'm missing here? Much appreciated!

import urllib.request, urllib.parse, urllib.error
import json
import csv
import ssl
import os

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

api_key = 'XXX'
puuid = 'XXX'

matchlist = ['0e8194de-f011-4018-aca2-36b1a749badd','ae207558-599e-480e-ae97-7b59f97ec8d7']

f = csv.writer(open('my_file.csv','w+'))

for matchid in matchlist: 
    matchdeturl = 'https://europe.api.riotgames.com/lor/match/v1/matches/'+ matchid +'?api_key=' + api_key
    matchdetuh = urllib.request.urlopen(matchdeturl, context = ctx)
    matchdet = json.loads(matchdetuh.read().decode())
    matchplayers = matchdet['info']

    #if file is blank, write headers, if not write values
    if os.stat('my_file.csv').st_size == 0:
        f.writerow(list(matchplayers))
        f.writerow(matchplayers.values())
    else:
        f.writerow(matchplayers.values())
LionelF
  • 23
  • 4

1 Answers1

1

It's possible that the file buffers instead of writing immediately to disk because IO is an expensive operation. Either flush the file before checking its size, or set a flag in your loop and check that flag instead of checking the file size.

f = csv.writer(open('my_file.csv','w+'))

needs_header = os.stat('my_file.csv').st_size == 0

for matchid in matchlist: 
    # do stuff

    #if file needs a header, write headers
    if needs_header:
        f.writerow(list(matchplayers))
        needs_header = False

    # Then, write values
    f.writerow(matchplayers.values())
Pranav Hosangadi
  • 23,755
  • 7
  • 44
  • 70
  • Thank you Pranav, that worked! Out of interest, how would you flush the file? My print(os.stat('my_file.csv').st_size) statement outside of the for loop still prints 0 after everything has been written correctly - I'm assuming this means the file is buffered and only written when the entire program is done executing? – LionelF Apr 09 '21 at 20:50
  • Ah! Well, this is why it's recommended to use `with` to handle files. When the code execution exits the `with` block, the file is automatically flushed and closed. In your case, I suggest instead of instantiating the csv writer and opening the file in the same statement, you do something like: `fh = open('my_file.csv', 'w+'); f = csv.writer(fh)`. Then you can do `fh.flush()` or `fh.close()` when you think it's necessary – Pranav Hosangadi Apr 09 '21 at 20:52
  • Super useful, did as you suggested and it worked perfectly. Thanks! – LionelF Apr 09 '21 at 21:04