0

I have been attempting to deploy a web scraper written in python for the past few weeks on azure. I initially tried to do this in an azure app service by building pushing a docker image to the service. I have had success previously with this method when deploying a flask rest api. Unfortunately the time out limit of the azure app service meant that the web scraper container is terminated as it is does not give a proper response when azure attempts to get a response.

I have since tried setting up a windows based app service in order to create an azure WebJob however, this has the problem of being capped at python 3.6 which i believe is causing import problems. I cannot import the "requests" module which is essential for the scraper to work correctly. I have a requirements.txt inside the zip i upload for the webjob but this doesnt seem to allow the importing of this module either. Is there a way to import modules from inside the webjob or is there another/better way to deploy a repetitive task on azure that doesnt have to give an api response like a normal azure app service deployment.

Below is the error i am receiving in the webjob logs :

[12/19/2020 16:36:48 > 658fd3: SYS INFO] Run script 'run.py' with script host - 'PythonScriptHost'
[12/19/2020 16:36:48 > 658fd3: SYS INFO] Status changed to Running
[12/19/2020 16:36:48 > 658fd3: ERR ] Traceback (most recent call last):
[12/19/2020 16:36:48 > 658fd3: ERR ]   File "run.py", line 12, in <module>
[12/19/2020 16:36:48 > 658fd3: ERR ]     import requests
[12/19/2020 16:36:48 > 658fd3: ERR ] ModuleNotFoundError: No module named 'requests'

Any help is greatly appreciated thank you.

Jono
  • 1

1 Answers1

1

You can use Azure's Serverless feature called Functions. If you want to make more use of it, then make a docker container with any image you want and upload it as your function, but beware of cold starts

Mohamed Abdou
  • 441
  • 3
  • 11