No logs in Apache Airflow with KubernetesExecutor in AWS EKS

Question

I managed to deploy Airflow 1.10.12 in AWS EKS. I have a webserver and a scheduler which run on KubernetesExecutor. When I run a DAG, I spawn a pod and run the dag inside it with LocalExecutor.

Unfortunately, I am not able to see the logs in the container. If I look at the container logs, all I see is:

[2020-10-14 14:03:42,438] {__init__.py:50} INFO - Using executor LocalExecutor
[2020-10-14 14:03:42,438] {dagbag.py:417} INFO - Filling up the DagBag from /opt/airflow/dags/repo/dags/logging_test.py
Running %s on host %s <TaskInstance: test_logging_dag.test_task 2020-10-14T14:03:31.985696+00:00 [queued]> testloggingdagtesttask-c1a70a2eb22a4533a7dfa797dc83c41b

The dag I am running is:

import logging

from airflow.models import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.utils.timezone import datetime
from time import sleep

logger = logging.getLogger(__name__)


def test_logging_fn(**kwargs):
    logger.info("Log from DAG Logger")
    kwargs["ti"].log.info("Log from TI Logger")
    print("Log from Print statement")
    sleep(360)


dag = DAG(
    dag_id='test_logging_dag',
    schedule_interval=None,
    start_date=datetime(2020, 10, 14)
)

PythonOperator(
    task_id='test_task',
    python_callable=test_logging_fn,
    provide_context=True,
    dag=dag,
)

My configmap with airflow.cfg is:

airflow.cfg: |-
    [api]
    auth_backend = airflow.api.auth.backend.deny_all
    
    [core]
    colored_console_log = False
    dags_folder = /opt/airflow/dags/repo/dags
    executor = KubernetesExecutor
    load_examples = False
    remote_logging = False

    [kubernetes]
    airflow_configmap = airflow-airflow-config
    airflow_local_settings_configmap = airflow-airflow-config
    multi_namespace_mode = False
    namespace = airflow
    pod_template_file = /opt/airflow/pod_templates/pod_template_file.yaml
    worker_container_repository = apache/airflow
    worker_container_tag = 1.10.12
    delete_worker_pods = False
    
    [logging]
    base_log_folder = /opt/airflow/logs
    logging_level = INFO
    
    [scheduler]
    run_duration = 41460
    scheduler_heartbeat_sec = 5
    statsd_host = airflow-statsd
    statsd_on = True
    statsd_port = 9125
    statsd_prefix = airflow
    
    [webserver]
    enable_proxy_fix = True
    expose_config = True
    rbac = True

I ssh into a running container and look in /opt/airflow/logs. I can see the .log file inside but it is empty.

Does anyone know why would this happen?

Thanks.