Can I use Airflow to start/stop spark streaming job

Question

I have two type of job: Spark Batch jobs and and Spark streaming jobs.

I would like to schedule and manage them both with airflow.

Airflow is using for job has stop. But I want to use it for my streaming job. Can anyone give me some idea or other tool for my streaming job

score 1 · Accepted Answer · answered Oct 28 '21 at 13:31

1

Batch jobs - including spark can be scheduled via airflow scheduler.

Streaming Jobs - technically yes - there is nothing stopping you. However, it would not make sense to do the same for spark streaming jobs as these are long running jobs and it would be better to execute them via other tools.

answered Oct 28 '21 at 13:31

floating_hammer

409
3
10

thanks for your answer. can you tell me some tool for streaming jobs – Hiệp Oct 29 '21 at 03:18

Can I use Airflow to start/stop spark streaming job

1 Answers1