3

My current project requires automation and scheduled execution of a number of tasks (copy a file, send an email when a new file arrives in a directory, execute an analytics job, etc). My plan is to write a number of individual shell scripts for each task but would need a workflow orchestration/job flow management tool.

I am aware of a number of workflow orchestration tools such as Airflow, Luigi, Azkaban and have used them in the past as well, but those seem to be designed for *nix systems and unless i am mistaken they don't support Windows.

What are some good tools and solutions to achieve this?

Praveen Thirukonda
  • 365
  • 1
  • 4
  • 16

1 Answers1

0

Brilliant guide for a Data Engineer https://github.com/pawl/awesome-etl

EDIT Apparently since adding gunicorn, airflow has not been compatible with windows.

Evan Kairuz
  • 76
  • 3
  • I am aware of Airflow and it definitely fits the bill and is great. It was actually my first preference as well, but after spending a lot of time trying to make it work on Windows 8, i have given up. Have you had success making it run on Windows? – Praveen Thirukonda Mar 02 '18 at 05:49
  • 1
    You're right about airflow on windows. Updated the answer to just include that list of alternatives. – Evan Kairuz Mar 02 '18 at 06:12