0

I have a fully functional django web app running in windows local machine. However, I now need to deploy it in aws ec2 windows server.

  1. This is "upload - process - download" type of application.
  2. since the processing is quite heavy, I want to shift it to databricks notebook.
  3. So, DB Notebook should access the input file, process it and later save the output which can be downloaded by using web app.

My question is,

  1. Can this be done ?

  2. I was thinking of a way, where I can trigger a notebook through rest API request with required parameters. ( I couldn't find any way ) If I trigger DB Notebook with AWS-lambda, then can I trigger AWS- lambda through rest API ?

  3. Both Input and Output can be saved to either DBFS / S3 ?

  4. If someone has worked on similar activity, can anyone suggest a way to do it.

I am quite new to databricks, thus not aware of most of it's functionalities.

  • Note - Both Input & Output are .csv format files. I understand this could be a similar to some other question here, I couldn't find such specific use case.
Chetan Jawale
  • 39
  • 9
  • Yes Databricks can read data from S3 and store it back in S3 You can trigger Databricks JOB via API. https://docs.databricks.com/dev-tools/api/latest/jobs.html – Ganesh Chandrasekaran Jun 03 '22 at 12:14
  • This is an older article, see whether it helps you https://databricks.com/blog/2016/10/11/using-aws-lambda-with-databricks-for-etl-automation-and-ml-model-serving.html – Ganesh Chandrasekaran Jun 03 '22 at 12:15

0 Answers0