Using .write() in Python only writes a single line

Question

So, as an assignment from Thenewboston, I'm trying to grab a block of code from his site and write it to a file. The code grabbing part works just fine, but the writing part doesn't work:

import requests
from bs4 import BeautifulSoup


def crawler(url):
    source = requests.get(url)
    source_text = source.text
    soup_obj = BeautifulSoup(source_text, "html.parser")
    for code in soup_obj.find('code'):
        codes = str(code)
        final_code = codes.replace('<br/>', '').replace('Â', '')
        print(final_code)
        fx = open('crawler_code.txt', 'w')
        fx.write(final_code)
        fx.close()

crawler('https://thenewboston.com/forum/topic.php?id=1610')

whats the error you are getting, please mention in question itself by editing — Sanket, Dec 04 '17 at 05:47

score 2 · Accepted Answer · answered Dec 04 '17 at 05:52

You're overwriting the file within the loop

Open the file for writing before you loop and write

with open('crawler_code.txt', 'w') as fx:
    for code in soup_obj.find('code'):
        codes = str(code)
        final_code = codes.replace('<br/>', '').replace('Â', '')
        print(final_code)
        fx.write(final_code)

score 0 · Answer 2 · answered Dec 04 '17 at 05:51

You did not really mentioned what was the issue, but I think as you open the file for each line you're grabbing (therefore erasing the content with 'w' mode), you only have one line written in the file at the end. Is it your problem ?

If so, try to open the file before your loop, at the beginning of the crawl method.

Using .write() in Python only writes a single line

2 Answers2