0

I'm trying to run my Differential Evolution script in Python called differential_evolution.py. Each iteration runs for around 40 generations. I want to run 50 iterations in parallel using Airflow. I have provided random seed in my script so that each iteration creates different results.

Snippet of differential_evolution.py:

Optimizer() is a custom class I created to run the algorithm. solution stores the solution list in list attribute x. And mape calculates the mape for the solution list x.

for iteration in range(50):
    seed = np.random.randint(0, 1000)  
    opt_obj = Optimizer()  
    solution = opt_obj.run_optimizer()  
    mape = opt_obj.calc_performance(solution.x)

Each iteration creates two output files: abc.txt and xyz.csv to store relevant information for different variables.

Snippet of the dag script:

start >> create_cluster >> differential_evolution.py >> delete_cluster >> end

This is running fine but taking lots of time when you run for 50 iterations.

What I want is to create a dag like this:

start >> create_cluster >> [iteration 1, iteration 2, ... iteration 50] >> delete_cluster >> end, where each iteration outputs the same two files abc_i.txt and xyz_i.csv (i is the ith iteration)

Sanket kumar
  • 1

1 Answers1

1

You could do something like this:

@task
def task1():
    something here

@task
def task2():
    something here

@task
def task_optimizer():
   seed = np.random.randint(0, 1000)  
   opt_obj = Optimizer()  
   solution = opt_obj.run_optimizer()  
   mape = opt_obj.calc_performance(solution.x)


start_task=task1()
end_task=task2()

for iteration in range(50):
    optimize_task = task_optimizer()
    start_task >> optimize_task >> end_task

You would need to adjust so the optimizer generates random numbers properly but something along this line should work.

Axe_83
  • 33
  • 8
  • This code will just run all the 50 iterations sequentially. I was wondering if they can be done in parallel. – Sanket kumar Jul 20 '22 at 01:49
  • I think you also need to reconfigure airflow. Is sequential or local executor enabled in your configuration file? You also need PostgreSQL instead of the default SQLite configured for this to work. Check this answer https://stackoverflow.com/questions/52741536/running-airflow-tasks-dags-in-parallel – Axe_83 Jul 20 '22 at 14:42