1

Currently I am trying to use Apache Airflow with Celery executor. For this I have Redis service from IBM cloud. This service has the TLS connection type which means it has the redis protocol as rediss://. Side not: I am using puckel's airflow dockerfile.

I have set redis parameters and my broker url is in the form of rediss://username:password@hostname:port/virtual_host. While I try to run for example, Flower I get these errors:

Secure redis scheme specified (rediss) with no ssl options, defaulting to insecure SSL behaviour.

Steps that I have done till now:

  • I have added the following lines to the entrypoint.sh:
    : "${AIRFLOW__CELERY__SSL_ACTIVE:="True"}"
    : "${AIRFLOW__CELERY__BROKER_USE_SSL:="{'ssl_cert_reqs': ssl.CERT_REQUIRED, 'ssl_ca_certs': '/6be25d73-0600-11ea-9bce-eaebe975ceba.crt'}"}"
    AIRFLOW__CELERY__BROKER_USE_SSL="${AIRFLOW__CELERY__BROKER_USE_SSL}"
    AIRFLOW__CELERY__SSL_ACTIVE="${AIRFLOW__CELERY__SSL_ACTIVE}"
    export AIRFLOW__CELERY__SSL_ACTIVE
    export AIRFLOW__CELERY__BROKER_USE_SSL
  • I have tried to use redis:// with the same broker url, but flower even did not started.
Ersoy
  • 8,816
  • 6
  • 34
  • 48
Altun Baratlı
  • 13
  • 4

1 Answers1

2

Yes it does, but you need a fairly recent version of Kombu and redis-py. We simply have a broker URL that looks like rediss://:BABADEDAuikxWx0oPZYfPE3IXJ9BVlSC@broker.example.com:6379/12?ssl_cert_reqs=CERT_OPTIONAL and it "just works".

DejanLekic
  • 18,787
  • 4
  • 46
  • 77
  • Hello DejanLekic. Thanks for your help! I tried your method. The error message somehow started not to showing up. However my flower is not opening. I guess sending TLS parameters over the url did not work for me. Have you also set SSL_ACTIVE parameter to TRUE? Thanks. – Altun Baratlı May 20 '20 at 07:56
  • I suppose Flower runs in a separate virtual environment - did you perhaps simply forget to install latest Kombu? Our Flower service uses encrypted communication to Redis and we do it as I described above. – DejanLekic May 20 '20 at 09:58