Prefect is a Python-based workflow management system (ETLs are an example use-case). Users organize Tasks into Flows, define dependencies, schedules, etc., and Prefect takes care of the rest.
Prefect
Prefect is an open-source workflow and orchestration framework, written in Python 3, that bills itself as an up-and-coming alternative to Airflow. Its design philosophy emphasizes the benefits of negative engineering: that is, features designed to manage the failure and recoverability of workflows as a natural extension of normal development. Its creators also tout the benefits of its hybrid execution model, whereby orchestration occurs with zero knowledge of either the code being run or the data being manipulated. It also boats features such as first-class workflow scheduling, dynamic task generation, and horizontal workflow scalability via out-of-the-box integration with Dask Distributed.
Prefect consists of three components:
- Prefect Core: the central features of development in the Prefect ecosystem, by which Tasks are composed into directed acyclic graphs (DAGs) called Flows.
- Prefect Server: the GraphQL application and UI that, taken together, allow users to manage flow submission and execution in an easy-to-use and interactive manner.
- Prefect Cloud: the optional commercial offering of the Prefect maintainers, which organizations can use to leverage managed infrastructure in addition to the benefits of Prefect Server.
More information about Prefect is available at:
