Airflow - Failed to fetch log file from worker. 404 Client Error: NOT FOUND for url

Question

I am running Airflowv1.9 with Celery Executor. I have 5 Airflow workers running in 5 different machines. Airflow scheduler is also running in one of these machines. I have copied the same airflow.cfg file across these 5 machines. I have daily workflows setup in different queues like DEV, QA etc. (each worker runs with an individual queue name) which are running fine.

While scheduling a DAG in one of the worker (no other DAG have been setup for this worker/machine previously), I am seeing the error in the 1st task and as a result downstream tasks are failing:

*** Log file isn't local.
*** Fetching here: http://<worker hostname>:8793/log/PDI_Incr_20190407_v2/checkBCWatermarkDt/2019-04-07T17:00:00/1.log
*** Failed to fetch log file from worker. 404 Client Error: NOT FOUND for url: http://<worker hostname>:8793/log/PDI_Incr_20190407_v2/checkBCWatermarkDt/2019-04-07T17:00:00/1.log

I have configured MySQL for storing the DAG metadata. When I checked task_instance table, I see proper hostnames are populated against the task.

I also checked the log location and found that the log is getting created.

airflow.cfg snippet:

base_log_folder = /var/log/airflow
base_url = http://<webserver ip>:8082
worker_log_server_port = 8793
api_client = airflow.api.client.local_client
endpoint_url = http://localhost:8080

What am I missing here? What configurations do I need to check additionally for resolving this issue?

Shouldnt the base url point to webserver ip ?You have mentioned it as scheduler ip address — naveen kumar, Aug 20 '19 at 10:35
Yeah the scheduler and webserver were running in the same node. Will update the airflow.cfg snippet accordingly. Thanks for pointing out. — riyaB, Aug 21 '19 at 06:54

score 6 · Answer 1 · answered Apr 21 '20 at 10:14

6

Looks like the worker's hostname is not being correctly resolved. Add a file hostname_resolver.py:

import os
import socket
import requests
def resolve():
    """
    Resolves Airflow external hostname for accessing logs on a worker
    """
    if 'AWS_REGION' in os.environ:
        # Return EC2 instance hostname:
        return requests.get(
            'http://169.254.169.254/latest/meta-data/local-ipv4').text
    # Use DNS request for finding out what's our external IP:
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    s.connect(('1.1.1.1', 53))
    external_ip = s.getsockname()[0]
    s.close()
    return external_ip

And export: AIRFLOW__CORE__HOSTNAME_CALLABLE=airflow.hostname_resolver:resolve

answered Apr 21 '20 at 10:14

dvainrub

133
2
6

1

Where to add this python file? I'm also facing the same issue and using CeleryExecutor with Mysql database on Airflow 2.0.0 – Nikita Mar 11 '21 at 05:55
@Nikita anywhere you want, just make sure you load it with your dockerfile – dvainrub Mar 12 '21 at 09:59
I'm not using docker. So can I place this in plugins directory? – Nikita Mar 22 '21 at 06:35
It’s better to create a new folder and not to mix it with the plugins – dvainrub Mar 23 '21 at 08:36
2

I created a config folder and put it there. For 2.1 the syntax was slightly changed to AIRFLOW__CORE__HOSTNAME_CALLABLE=airflow.hostname_resolver.resolve – Ivailo Bardarov Mar 29 '21 at 15:36
I updated the files and exported the value but looks same issue in version 2.02. – PANDURANG BHADANGE May 18 '21 at 05:49

score 0 · Answer 2 · answered Oct 11 '21 at 02:21

The web program of the master needs to go to the worker to fetch the log and display it on the front-end page. This process is to find the host name of the worker. Obviously, the host name cannot be found,Therefore, add the host name to IP mapping on the master's vim /etc/hosts

score 0 · Answer 3 · answered Dec 23 '22 at 15:07

If this happens as part of a Docker Compose Airflow setup, the hostname resolution needs to be passed to the container hosting the webserver, e.g. through extra_hosts:

# docker-compose.yml

version: "3.9"

services:
  webserver:
    extra_hosts:
      - "worker_hostname_0:192.168.xxx.yyy"
      - "worker_hostname_1:192.168.xxx.zzz"
    ...
  ...

More details here.

Airflow - Failed to fetch log file from worker. 404 Client Error: NOT FOUND for url

3 Answers3