6

Currently i am using Airflow with Version : 1.10.10

After opening into airflow/logs folder there are many folder that are named based on your DAG name but there is a folder named scheduler which when opened consist folder that are named in date format ( E.g 2020/07/08 ) and it goes until the date when i first using airflow.After searching through multiple forum I'm still not sure what this folder logs are for.

Anyway the probelm is I kept wondering if it is okay to delete the contents inside scheduler folder since it takes so much space unlike the rest of the folder that are named based on the DAG name (I'm assuming thats where the log of each DAG runs is stored). Will the action of deleting the contents of scheduler cause any error or loss of DAG log?.

This might be a silly question but i want to make sure since the Airflow is in production server. I've tried creating an Airflow instance in local instance and delete the scheduler folder contents and it seems no error have occurred. Any feedback and sharing experience on handling this issue is welcomed

Thanks in Advance

Anindhito Irmandharu
  • 135
  • 1
  • 8

1 Answers1

9

It contains the logs of airflow scheduler afaik. I have used it only one time for a problem about SLAs.

I've been deleting old files in it for over a year, never encountered a problem. this is my command to delete old log files of scheduler:

find /etc/airflow/logs/scheduler -type f -mtime +45 -delete
mustafagok
  • 1,672
  • 10
  • 15
  • “ I have used it only one time for a problem about SLAs.“, May i know the detail of your problem with SLA?. Anyway thanks for the answer and my apology for replying so late. – Anindhito Irmandharu Jul 12 '20 at 05:26
  • SLA is for scheduled dags, I used it with a non-scheduled dag (`schedule_interval=None`). My dag looks like normal, rendering without problem (not broken, and no error at all) but tasks never scheduled. I find out that from scheduler logs, there was an error about types which is raised from some SLA method of airflow. – mustafagok Jul 14 '20 at 13:33
  • Oh i see maybe because i've always used schedule interval i have not encountered problem like you. Ty for the explanations. – Anindhito Irmandharu Jul 17 '20 at 05:00