0

I seem to come across unique situations throughout this month.

I need to read a file from the web and update the database.

I have taken two approaches :

Approach 1.

Upload the values from the application itself - this could be time consuming and I could hit worker timeout - however, I have retained this approach - as it currently takes 27-29 seconds. Heroku timeout is at 30 seconds.

Approach 2.

Upload the values from Django Command Management . I have scheduled a job via Heroku.

I used pandas - pd.read_csv file for reading a file from the web in both approaches.

Scenario 1:

If I use approach 1,

Localhost : It works fine

Heroku : It works fine

Scenario 2 :

If I use approach 2,

Localhost : It works fine

Heroku : HTTP 503 error and not able to read the file

What could be the solution for this ?

P.S. - I plan to put requests and add a browser header and update this thread.

Community
  • 1
  • 1
Mani
  • 43
  • 7

1 Answers1

0

Found an answer to this , I shall confirm this tomorrow when the Heroku job runs.

Edited: Job worked perfectly fine, all data got updated. You have got to fake it :

Edited - 2nd edit, it failed the very next day. I am looking at generating few more fake headers and will post an update.

    from zipfile import BadZipFile, ZipFile
    import datetime
    import pandas as pd
    import logging
    import requests,zipfile
    import io

    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
    r = requests.get(url,headers=headers)

    logger.info("Status code = ",r.status_code)

    if r.status_code == 200:
        logger.info(" Request ran successfully ")
        file = zipfile.ZipFile(io.BytesIO(r.content))
        myfile = file.open(filename)
        stock_prices = pd.read_csv(myfile)
        print(stock_prices)
Mani
  • 43
  • 7