0

I am trying to enable Remote Airflow logs, to do that I followed these steps:

apache-airflow install

pip install apache-airflow[crypto,postgres,ssh,s3,log]==1.10.10

Airflow.cfg file:

remote_logging = True
remote_base_log_folder = s3://bucket-name/logs
encrypt_s3_logs = False
logging_level = INFO
fab_logging_level = WARN
remote_log_conn_id = MyS3Conn

I have Airflow running in a docker container in an AWS ECS Blue/Green deploy. I read that if airflow is hosted on an EC2 server you should create the connection leaving everything blank in the configuration apart from connection type which should stay as S3.

The S3hook will default to boto and this will default to the role of the EC2 server you are running airflow on. Assuming this role has rights to S3 your task will be able to access the bucket.

So I applied this, but I don't know if using docker it works as intended.

If I run a dag I see the logs which are createds in the /urs/local/airflow/logs folder in the docker container, but there is no new files in the specified folder in S3.

mrc
  • 2,845
  • 8
  • 39
  • 73
  • If you use Docker, you are not really using your EC2, but the docker container, to execute Airflow. Thus, what you are saying won't work. Do you have in your docker-compose or Dockerfile any line to include the files aws_credentials and aws_config_file? – Javier Lopez Tomas Sep 05 '20 at 13:50
  • @JavierLópezTomás Yes I have that file, and boto3 is able to do some actions with these key and secret by default as I can see for other processes. So, the connection as its setup should work because they are detected by other processes. – mrc Sep 07 '20 at 09:24

0 Answers0