How to write to CSV asynchronously

Asked Jan 13 '18 at 19:15

Active Jan 13 '18 at 19:15

Viewed 1,156 times

I have to test about one million possible urls. If the url is valid I want to store it in a CSV. To speed things up I want to use asynchronously. Code works fine when ran with a for loop, but the CSV is empty if I run it asynchronously.

    #Declare CSV
import csv
csvFile = open('example.tsv', 'w', newline='')
    csvWriter = csv.writer(csvFile, delimiter='\t', lineterminator='\n\n')

    def parse(url):
        #get url's <title>
        data = getTitle(url)
        #if record is found

        if(data.title.string != "RECORD INACTIVE" or data.title.string != "FOUND"):
            csvWriter.writerow([url])


    with ProcessPoolExecutor(max_workers=4) as executor:
        {executor.submit(parse, url): url for url in URLS}

When I run this Asunchronously the CSV file is empty.

But it works if I run it with a fro loop :

for url in URLS:
    parse(ul)

asked Jan 13 '18 at 19:15

Lev

1

Are you providing a call back? – Mike Tung Jan 13 '18 at 19:16
@MikeTung no I am trying to write to the CSV inside the parse function. – Lev Jan 13 '18 at 19:18
1

use one file per thread, merge files later? its just text? and you avoid some file locking hassle? – Patrick Artner Jan 13 '18 at 19:22
2

Can you clearly provide the code for how you are calling it async - also did you mean to type: parse(url) in your for loop example above? – Bob Aleena Jan 13 '18 at 19:56

How to write to CSV asynchronously

0 Answers0