File Operation in Python

Question

What I am trying to do:

I am trying to use 'Open' in python and this is the script I am trying to execute. I am trying to give "restaurant name" as input and a file gets saved (reviews.txt).

Script: (in short, the script goes to a page and scrapes the reviews)

from bs4 import BeautifulSoup
from urllib import urlopen
queries = 0
while queries <201:
    stringQ = str(queries)
    page = urlopen('http://www.yelp.com/biz/madison-square-park-new-york?start=' + stringQ)

    soup = BeautifulSoup(page)
    reviews = soup.findAll('p', attrs={'itemprop':'description'})
    authors = soup.findAll('span', attrs={'itemprop':'author'})

    flag = True
    indexOf = 1
    for review in reviews:
        dirtyEntry = str(review)
        while dirtyEntry.index('<') != -1:
            indexOf = dirtyEntry.index('<')
            endOf = dirtyEntry.index('>')
            if flag:
                dirtyEntry = dirtyEntry[endOf+1:]
                flag = False
            else:
                if(endOf+1 == len(dirtyEntry)):
                    cleanEntry = dirtyEntry[0:indexOf]
                    break
                else:
                    dirtyEntry = dirtyEntry[0:indexOf]+dirtyEntry[endOf+1:]
        f=open("reviews.txt", "a")
        f.write(cleanEntry)
        f.write("\n")
        f.close

    queries = queries + 40

Problem: It's using append mode 'a' and according to documentation, 'w' is the write mode where it overwrites. When i change it to 'w' nothing happens.

f=open("reviews.txt", "w") #does not work!

Actual Question: EDIT: Let me clear the confusion.

I just want ONE review.txt file with all the reviews. Everytime I run the script, I want the script to overwrite the existing review.txt with new reviews according to my input.

Thank you,

Answer 1

If I understand properly what behavior you want, then this should be the right code:

with open("reviews.txt", "w") as f:
    for review in reviews:
        dirtyEntry = str(review)
        while dirtyEntry.index('<') != -1:
            indexOf = dirtyEntry.index('<')
            endOf = dirtyEntry.index('>')
            if flag:
                dirtyEntry = dirtyEntry[endOf+1:]
                flag = False
            else:
                if(endOf+1 == len(dirtyEntry)):
                    cleanEntry = dirtyEntry[0:indexOf]
                    break
                else:
                    dirtyEntry = dirtyEntry[0:indexOf]+dirtyEntry[endOf+1:]
        f.write(cleanEntry)
        f.write("\n")

This will open the file for writing only once and will write all the entries to it. Otherwise, if it's nested in for loop, the file is opened for each review and thus overwritten by the next review.

with statement ensures that when the program quits the block, the file will be closed. It also makes code easier to read.

---

I'd also suggest to avoid using brackets in if statement, so instead of

if(endOf+1 == len(dirtyEntry)):

it's better to use just

if endOf + 1 == len(dirtyEntry):

Answer 2

If you want to write every record to a different new file, you must name it differently, because this way you are always overwritting your old data with new data, and you are left only with the latest record.

You could increment your filename like this:

# at the beginning, above the loop:

i=1

f=open("reviews_{0}.txt".format(i), "a")
        f.write(cleanEntry)
        f.write("\n")
        f.close
i+=1

---

UPDATE

According to your recent update, I see that this is not what you want. To achieve what you want, you just need to move f=open("reviews.txt", "w") and f.close() outside of the for loop. That way, you won't be opening it multiple times inside a loop, every time overwriting your previous entries:

f=open("reviews.txt", "w")
for review in reviews:
        # ... other code here ... #

        f.write(cleanEntry)
        f.write("\n")
f.close()

But, I encourage you to use with open("reviews.txt", "w") as described in Alexey's answer.