10

I started Airflow with no FERNET_KEY. Once I realised it, I did the following: https://airflow.apache.org/configuration.html#connections

pip install apache-airflow[crypto]

from cryptography.fernet import Fernet
fernet_key= Fernet.generate_key()
print(fernet_key)

took the key and placed it in airflow.cfg and then called airflow initdb, but the error still appears.

What am I doing wrong?

When I do:

airflow webserver -D

I get:

  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 713, in extra_dejson
    if self.extra:
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 293, in __get__
    return self.descriptor.__get__(instance, owner)
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 632, in get_extra
    return fernet.decrypt(bytes(self._extra, 'utf-8')).decode()
  File "/usr/lib/python2.7/dist-packages/cryptography/fernet.py", line 101, in decrypt
    raise InvalidToken

The log indicates that there is issue with this code:

def get_conn(conn_id, session=None):
    conn = (session.query(Connection)
                   .filter(Connection.conn_id == conn_id)
                   .first())
    return conn


def my_python_function():
   conn = get_conn('s3connection')
   key_id = conn.extra_dejson.get('aws_access_key_id')
   secret_key = conn.extra_dejson.get('aws_secret_access_key')
   default_region = conn.extra_dejson.get('region_name')
   return key_id,secret_key,default_region
Simon
  • 5,464
  • 6
  • 49
  • 85
jack
  • 821
  • 5
  • 16
  • 28
  • Did you make any changes to the fernet key after the first time you added it to the airflow.cfg? – cwurtz Jun 13 '18 at 14:06
  • @CJWurtz This is the first DAG I'm lunching. After I installed airflow and uploaded the DAG I got a similar message. Then I made the modification listed in the question. – jack Jun 13 '18 at 14:33
  • @tobi6 the doc says "If crypto package was not installed initially, you can still enable encryption for connections by following steps below" so there is a way to solve it. After i installed it I deleted the connections and recreate them. Still didn't work. What else can I do? – jack Jun 13 '18 at 14:36
  • @tobi6 no quotes.. Actualy there was a previous key there (dont know from where) I simply replaced it. No quotes and no single quotes. – jack Jun 13 '18 at 14:39
  • @tobi6 The issue is that section 6 says "Restart AirFlow webserver." But I can't restart the server because of the python errror – jack Jun 13 '18 at 14:43
  • @tobi6 didn't change any encoding. Again, I created a key only once.. after the error appeared... But the cfg file had a value there before. – jack Jun 13 '18 at 14:50

5 Answers5

6

Airflow usually generates one for you.

Here's an example:

$ python
>>> from cryptography.fernet import Fernet
>>> k=Fernet.generate_key()
>>> print(k)
Z6BkzaWcF7r5cC-VMAumjpBpudSyjGskQ0ObquGJhG0=
>>> ^D
$ $EDITOR $AIRFLOW_HOME/airflow.cfg

There change:

# Secret key to save connection passwords in the db
fernet_key = cryptography_not_found_storing_passwords_in_plain_text

to:

# Secret key to save connection passwords in the db
fernet_key = Z6BkzaWcF7r5cC-VMAumjpBpudSyjGskQ0ObquGJhG0=

Check if it's set as expected (or it'll generate a random one each time)

$ python
Python 2.7.13 (default, Jul 18 2017, 09:17:00)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from airflow import configuration as conf
[2018-06-14 17:53:36,200] {__init__.py:57} INFO - Using executor SequentialExecutor
>>> conf.get('core','fernet_key')
'Z6BkzaWcF7r5cC-VMAumjpBpudSyjGskQ0ObquGJhG0='
>>>

The above should be v1.9.0 & v1.8.2 syntax [fixed], I've double checked this with the latter.

Whenever you change your fernet key, you need to delete all the connections and variables that are using encryption, as they won't decrypt anymore.

You could reset your db, but that's probably overdoing it.

dlamblin
  • 43,965
  • 20
  • 101
  • 140
  • When I do `c=AirflowConfigParser(default_config=AIRFLOW_CONFIG)` I get : `TypeError: __init__() got an unexpected keyword argument 'default_config'` – jack Jun 14 '18 at 07:54
  • Also $EDITOR $AIRFLOW_HOME/airflow.cfg didn't work for me. I had to do cd airflow and then nano airflow.cfg – jack Jun 14 '18 at 07:55
  • @jack Yes, you can choose your editor. If you read `man man` and `man crontab` you'll see that posix recommends to set EDITOR equal to the path to your favorite editor. My mistake on the default_config, that's post v1.9.0. So actually you want the updates I made above. – dlamblin Jun 14 '18 at 08:53
  • Works with 1.10 as well. Thank you! – Leo Oct 22 '18 at 12:43
6

Using Fernet, Airflow encrypt all the passwords for its connections in the backend database.

In your case, Airflow backend is using previous fernet key and you have generated a key using which you have created new connection.

My recommendation is to do the following First,

airflow resetdb

this will help in deleting all the existing records in your backend db.

Then,

airflow initdb

this will initialize backend like fresh.

Then start airflow web server and scheduler

airflow web server -p {port}
airflow scheduler

Then create new connection of s3 in UI (in extra - {"aws_access_key_id":"_your_aws_access_key_id_", "aws_secret_access_key": "_your_aws_secret_access_key_"})

Now you should be able to test s3 file watcher by following solution from - Airflow s3 connection using UI

shyam
  • 537
  • 6
  • 16
2

This can also happen if you have the scheduler and webserver running in separate Docker containers without a shared config. Each container will generate its own fernet key meaning they cannot decode each other's secrets.

In this case you could:

  • use the AIRFLOW__CORE__FERNET_KEY env var to explicitly set the key for both containers
  • set up a volume to share the config file between containers
  • run both processes in one container
Tamlyn
  • 22,122
  • 12
  • 111
  • 127
0

If you started airflow without a fernet key, it wouldn't allow any connection to be encrypted.

If you had a fernet key, added/edited any connections, they will be encrypted with that key. If you change the key in any way, those connections won't be decryptable. If you currently have connections encrypted with different fernet keys the only solution is migrating the keys created with one to the other. Or just delete them and recreate them assuming you still have the values handy.

cwurtz
  • 3,177
  • 1
  • 15
  • 15
  • I have only 1 connection. I deleted it and start over... But still - it doesn't work. There is a check on Is Extra Encrypted but not on Is Encrypted – jack Jun 13 '18 at 14:52
  • CJ if what you say would have been the solution then deleting all connections and then define them again should have been solve it. But it doesn't. Also I don't understand what exactly is encrypted... I think the encryption itself doesn't work so the decryption doesn't work either. – jack Jun 13 '18 at 15:06
  • The password field and the extra field are each encrypted separately. If `Is Encrypted` isn't checked and `Extra Ecnrypted` is, that means the password isn't encrypted, but the extras is encrypted. Encryption in airflow definitely works. – cwurtz Jun 13 '18 at 15:42
  • Perhaps try a very simple fernet key, like "testing", add a connection using it and see if you are getting the same error. Perhaps it's an issue with some character of the fernet key itself that you are using. – cwurtz Jun 13 '18 at 15:43
  • I didn't do the "EXPORT AIRFLOW__CORE__FERNET_KEY = your_fernet_key" because the doc said " Alternatively, " could this be it? – jack Jun 13 '18 at 15:44
  • In airflow, when it attempts to get a configuration variable, it will first check for an ENV var, followed up with the airflow.cnf, then command args, and lastly defaults. You don't need to set it by exporting a ENV var if you have it in your airflow.cfg. – cwurtz Jun 13 '18 at 15:48
  • honestly I do not understand this problem. I created a new key. deleted all connections. saved them again. Why this keeps happening?! I see that the module returns: return Fernet(configuration.get('core', 'FERNET_KEY').encode('utf-8')) https://airflow.apache.org/_modules/airflow/models.html how can I check why this line isn't working? – jack Jun 13 '18 at 15:50
  • Easiest way to check would be grab the value of the connection from database, open a python console window, copy and paste the `get_fernet()` method without the `AirflowException` (or import that too), and attempt to run the fernet decode with the value from the db. Just run through each command step by step and see where something unexpected happens – cwurtz Jun 13 '18 at 16:11
  • NameError: name 'get_fernet' is not defined – jack Jun 13 '18 at 16:14
  • You'll want to copy and paste it from airflow's models.py taking the part you need to create a Fernet object – cwurtz Jun 13 '18 at 16:18
  • I have the FERNET object. now what? – jack Jun 13 '18 at 16:23
  • I said copy and paste, so you can modify the funciton if needed to debug. However if you go that route, you need to do `from airflow.models import get_fernet`. Basically just trace the route airflow goes through when it runs `get_extra()`, passing in what you need to decrypt from the raw value from the db – cwurtz Jun 13 '18 at 16:23
  • When I do: print Fernet(configuration.get('core', 'FERNET_KEY').encode('utf-8')).__encryption_key I see: "▒▒0▒8▒▒[Ƞ/I▒9 – jack Jun 13 '18 at 16:25
  • The issue may be in the encryption itself? When I check the UI Connections I don't see anything encrypted there. – jack Jun 13 '18 at 16:43
  • If `configuration.get('core', 'FERNET_KEY').encode('utf-8')` matches what you have configured it should be correct then. Not sure if stackoverflow is formatting the characters of not, I did the same thing locally and have `b'\xad\x9e9l\x97\xe9\xd4\x1fF4Y"\t\xfcLj'`. I'd proceed with trying to use your Fernet object to decode a value from the db – cwurtz Jun 13 '18 at 16:43
0

Make sure to place fernet key first and then execute the following command:

airflow initdb

If issue still persist then delete all the tables from database and then rerun the command:

airflow initdb
Ankit Adlakha
  • 1,436
  • 12
  • 15