Adding logs to Airflow Logs

Question

How can I add my own logs onto the Apache Airflow logs that are automatically generated? any print statements wont get logged in there, so I was wondering how I can add my logs so that it shows up on the UI as well?

score 25 · Accepted Answer · answered Jan 25 '17 at 00:57

25

I think you can work around this by using the logging module and trusting the configuration to Airflow.

Something like:

import ...

dag = ...

def print_params_fn(**kwargs):
    import logging
    logging.info(kwargs)
    return None

print_params = PythonOperator(task_id="print_params",
                              python_callable=print_params_fn,
                              provide_context=True,
                              dag=dag)

answered Jan 25 '17 at 00:57

Ivan Gozali

2,089
1
27
25

2

I still just see `*** Reading local file: /home/ubuntu/airflow/logs///2019-08-30T17:26:27.272086+00:00/1.log` and none of the things I included in my logging statement. – benten Aug 30 '19 at 17:28
@benten after following the suggestion in this solution, I also can't see what I have included in my logging messages. Did you figure out how to get the logger to work? – lv10 Oct 20 '20 at 20:21

score 15 · Answer 2 · answered Dec 31 '19 at 08:18

15

Inside python callable for PythonOperator you can use:

import logging

LOGGER = logging.getLogger("airflow.task")
LOGGER.info("airflow.task >>> 2 - INFO logger test")

This will produce correct output like:

[2019-12-26 09:42:55,813] {operations.py:86} INFO - airflow.task >>> 2 - INFO logger test

For case with your custom logger:

LOGGER = logging.getLogger(__name__)
LOGGER.info("__name__ >>> 2 - INFO logger test")

You'll get duplication of formatting:

[2019-12-26 09:42:55,813] {logging_mixin.py:112} INFO - [2019-12-26 09:42:55,813] {operations.py:79} INFO - __name__ >>> 2 - INFO logger test

answered Dec 31 '19 at 08:18

Alexander K

151
1
3

1

How does this work with python modules? For example, if my callable calls a function from a module that I wrote and imported, how should the logging in that module be set up? Right now I call logging.getLogger(), and I get the duplicated formatting you describe. – Egahn May 14 '20 at 01:09
For python modules you can still use the `logging.getLogger("airflow.task")`. For a fix of the issue with the duplicates, check this answer: https://stackoverflow.com/a/73081021/4137497 – tsveti_iko Jul 22 '22 at 13:25

score 4 · Answer 3 · answered Oct 24 '16 at 21:05

If you look at the PythonOperator: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/python_operator.py#L80-L81, looks like there is no way to log STDOUT/STDERR from the python callable into the airflow logs.

However, if you look at the BashOperator: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/bash_operator.py#L79-L94, the STDOUT/STDERR from there is logged along with the airflow logs. So, if logs are important to you, I suggest adding the python code in a separate file and calling it using the BashOperator.

You can just `import logging` in Python and then do `logging.info('whatever logs you want')` and that will write to the Airflow logs. You don't need to invoke your Python code through the BashOperator just use the PythonOperator. This is shown in the above answer by Ivan Gozali which was written after this answer :) — Kyle Bridenstine, Sep 17 '18 at 18:06

Adding logs to Airflow Logs

3 Answers3

Linked