0

I use Python-Scrapy and Scrapinghub.

In my spider I should read and write a file

data_directory  = 'tmp'
csv_magasin = data_directory+"/"+current_script+"_"+current_date+"-shop_url.csv"

if not os.path.exists(data_directory):
    os.makedirs(data_directory)
try:
    os.remove('products.db')
except:
    pass
f = open('products.db', 'w')
f.write(t.content)
f.close
con = sqlite3.connect("products.db")
c = con.cursor()

When I run my spider on my server it works, but on Scrapinghub i can't read or write any file.

I read the doc but i didn't find how can i write a file in Scrapinghub. My question is what is the correct way to write a file on Scrapinghub

parik
  • 2,313
  • 12
  • 39
  • 67
  • Files on Scrapinghub are ephemeral and don't survive the job lifetime, what do you use the sqlite db for? – Pablo Hoffman Jan 25 '18 at 19:17
  • @PabloHoffman hey Pablo, i should delete my question because the way that i write the file is correct, 2 days ago, scrapinghub had a problem and i though that it's me, but it's was a technical problem of scrpainghub – parik Jan 26 '18 at 09:55

0 Answers0