0

I am trying to ship all airflow logs to kafka by attaching a new handler to the root logger, but not all logs are being published. Do I need to configure something else here?

This is what I'm doing:

custom_log_config.py

LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)

# Configure a new handler for publishing logs to kafka
environment = get_app_env()
LOGGING_CONFIG["handlers"]["kafka_handler"] = {
    "class": "com.test.log_handler.KafkaHandler",
    "formatter": "airflow",
    "version": environment.version,
    "log_file": log_file,
    "filters": ["mask_secrets"],
}

# Attach handler to root logger of airflow
LOGGING_CONFIG["root"]["handlers"].append("kafka_handler")

And finally I'm setting airflow configs to use the new logger class described above:

airflow.logging__logging_config_class=com.test.log_handler.custom_log_config.LOGGING_CONFIG

While some logs do flow to kafka, I'm missing task run logs (eg. following loggers: taskinstance.py, standard_task_runner.py, cli_action_loggers.py)

Aman
  • 115
  • 1
  • 10

1 Answers1

0

Look at the DEFAULT_LOGGING_CONFIG:

    'loggers': {
        'airflow.processor': {
            'handlers': ['processor'],
            'level': LOG_LEVEL,
            'propagate': False,
        },
        'airflow.task': {
            'handlers': ['task'],
            'level': LOG_LEVEL,
            'propagate': False,
            'filters': ['mask_secrets'],
        },
        'flask_appbuilder': {
            'handlers': ['console'],
            'level': FAB_LOG_LEVEL,
            'propagate': True,
        },
    },

You will find that tasks have separate logger "airflow.task"

Jarek Potiuk
  • 19,317
  • 2
  • 60
  • 61
  • Yes, I see there's a separate logger for tasks, but I'm attaching Kafka handler directly to the root logger. Any log request after going through airflow.task would also end up hitting root logger right? I wanted to understand why that's not happening. – Aman Sep 06 '22 at 09:13
  • 1
    No. Because the logger has prohttps://docs.python.org/3/library/logging.html#logging.Logger.propagate set to False. look at 'propagate': False, – Jarek Potiuk Sep 06 '22 at 15:27
  • Ah I missed that. Attaching handler at logger level worked :) – Aman Sep 06 '22 at 18:09
  • Is there a way to set different log level for different handlers under the same logger? I could use handler.set_level() if I was creating handler and then attaching to config. But instead I'm attaching class name, and initialization params, and that unfortunately doesn't accept log level today. – Aman Sep 21 '22 at 17:36
  • look for "advanced logging" in the Airflow docs – Jarek Potiuk Sep 22 '22 at 08:39