0

I'm running Aiflow 1.9 I ran a DAG that execute a BashOperator

bash_command="python /home/ubuntu/airflow/scripts/b.py "

Everything works great however when I go to the log of my task I see these:

[2018-06-17 08:00:10,273] {base_task_runner.py:98} INFO - Subtask: [2018-06-17 08:00:10,272] {bash_operator.py:101} INFO - log4j:WARN No appenders could be found for logger (com.amazonaws.athena.jdbc.AthenaDriver).
[2018-06-17 08:00:10,273] {base_task_runner.py:98} INFO - Subtask: [2018-06-17 08:00:10,273] {bash_operator.py:101} INFO - log4j:WARN Please initialize the log4j system properly.
[2018-06-17 08:00:10,273] {base_task_runner.py:98} INFO - Subtask: [2018-06-17 08:00:10,273] {bash_operator.py:101} INFO - log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

Seems like an unhandled issue by Airflow Apache?

jack
  • 821
  • 5
  • 16
  • 28

1 Answers1

1

Have you tried generating a custom log4j XML configuration file and point to it directly?

You may find this answer helpful.

EDIT: As seen in other issues, the warning appears related to AWS Athena JDBC. Airflow logger is probably verbose enough to pick it up, but the warning itself has nothing to do with it.

Michele 'Ubik' De Simoni
  • 730
  • 7
  • 19
  • I didn't even know what log4j is. I installed Airflow, configure it according to the guide and ran the script. The airflow documentation doesn't even mention log4j. This warning is created by Airflow not by my script. – jack Jun 18 '18 at 07:39
  • Are you sure that the logs are not related to the fact that you are trying to connect to AWS Athena? If I am not mistaken log4j is used by Athena JDBC. [Here](http://rstudio-pubs-static.s3.amazonaws.com/348157_191a890356fe493d98746eff129d2d7c.html) you can see the same warning in a different context, the only constant is the use of AWS Athena. – Michele 'Ubik' De Simoni Jun 18 '18 at 07:57
  • I am trying to connect to AWS Athena. This is what my script does. However when I run it manually I don't get such warning. When Airflow runs it for me - I get the warning. – jack Jun 18 '18 at 08:01
  • Either Airflow cannot access the configuration for some reason, or most likely it could be a verbosity issue. – Michele 'Ubik' De Simoni Jun 18 '18 at 08:15
  • But it can. The script works. It just shows warring. I am assuming this is a bug in Airflow. How can we report this issue? – jack Jun 18 '18 at 08:20
  • You can open an issue [here](https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-2320?filter=allopenissues). Before doing that make sure that is not a verbosity of output issue, it may very well be that when you manually run the script, the warning gets suppressed, but Airflow logger picks it up anyway. – Michele 'Ubik' De Simoni Jun 18 '18 at 08:25
  • OK. if we first assume that this is my script. The script is Python. What is the required fix? I have no XML configuration file whatsoever – jack Jun 18 '18 at 08:43
  • This comments line is getting pretty long, I would say let's discuss it in chat, but I have never worked with log4j, however having seen the same warning happening all over the place I am sure it is not an Airflow bug thus warranting its own question. – Michele 'Ubik' De Simoni Jun 18 '18 at 09:09
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/173395/discussion-between-michele-ubik-de-simoni-and-jack). – Michele 'Ubik' De Simoni Jun 19 '18 at 12:07