Airflow: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set

Question

I've setup airflow 2.0 on my local mac, backed with PostgresSQL for sql_alchemy_conn. My DAG's are running fine + maintaining metadata in sqla database, but any airflow users command errors with roles table/views not created.

I've tried setting and resetting airflow resetdb, but these tables are still not available in sql_alchemy_conn DB.

psycopg2.ProgrammingError: relation "ab_permission_view_role" does not exist
LINE 2: FROM ab_permission_view_role JOIN ab_permission_view ON ab_p...

[2019-01-11 11:45:58,223] {__init__.py:51} INFO - Using executor SequentialExecutor
/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/flask_sqlalchemy/__init__.py:774: UserWarning: Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. Defaulting SQLALCHEMY_DATABASE_URI to "sqlite:///:memory:".
  'Neither SQLALCHEMY_DATABASE_URI nor SQLALCHEMY_BINDS is set. '
Please make sure to build the frontend in static/ directory and restart the server
[2019-01-11 11:45:58,450] {manager.py:525} WARNING - No user yet created, use fabmanager command to do it.
[2019-01-11 11:45:59,117] {__init__.py:286} INFO - Filling up the DagBag from /Users/deepaksaroha/airflow/dags
[2019-01-11 11:46:00,114] {security.py:435} INFO - Start syncing user roles.
[2019-01-11 11:46:00,149] {security.py:185} INFO - Initializing permissions for role:Viewer in the database.
[2019-01-11 11:46:00,274] {security.py:185} INFO - Initializing permissions for role:User in the database.
[2019-01-11 11:46:00,397] {security.py:185} INFO - Initializing permissions for role:Op in the database.
[2019-01-11 11:46:00,487] {security.py:344} INFO - Fetching a set of all permission, view_menu from FAB meta-table
Traceback (most recent call last):
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
psycopg2.ProgrammingError: relation "ab_permission_view_role" does not exist
LINE 2: FROM ab_permission_view_role JOIN ab_permission_view ON ab_p...
             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/bin/airflow", line 32, in <module>
    args.func(args)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
    return f(*args, **kwargs)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/bin/cli.py", line 1353, in users
    appbuilder = cached_appbuilder()
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 214, in cached_appbuilder
    cached_app(config, testing)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 207, in cached_app
    app, _ = create_app(config, session, testing)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/app.py", line 167, in create_app
    security_manager.sync_roles()
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/security.py", line 443, in sync_roles
    self.create_custom_dag_permission_view()
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/security.py", line 391, in create_custom_dag_permission_view
    all_perm_views = set([role.permission_view_id for role in all_perm_view_by_user])
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 2995, in __iter__
    return self._execute_and_instances(context)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3018, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
    return meth(self, multiparams, params)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
    context)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
    exc_info
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
    raise value.with_traceback(tb)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
    context)
  File "/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "ab_permission_view_role" does not exist
LINE 2: FROM ab_permission_view_role JOIN ab_permission_view ON ab_p...
             ^
 [SQL: 'SELECT ab_permission_view_role.id AS ab_permission_view_role_id, ab_permission_view_role.permission_view_id AS ab_permission_view_role_permission_view_id, ab_permission_view_role.role_id AS ab_permission_view_role_role_id \nFROM ab_permission_view_role JOIN ab_permission_view ON ab_permission_view.id = ab_permission_view_role.permission_view_id JOIN ab_view_menu ON ab_view_menu.id = ab_permission_view.view_menu_id \nWHERE ab_permission_view_role.role_id = %(role_id_1)s AND ab_permission_view.view_menu_id != %(view_menu_id_1)s'] [parameters: {'role_id_1': 4, 'view_menu_id_1': 51}] (Background on this error at: http://sqlalche.me/e/f405)

I'm expecting that these tables should be created somewhere at the backend. I want to automate creating users through a script using airflow users command, but with existing errors, God save me :)

Answer 1

I got this solution'ed. This happened because ab_* tables were not created at airflow initdb. All these tables are for Role-based-access-control – RBAC.

To have these tables, follow the instructions:

1. edit airflow.cfg

[webserver]
rbac = True

2. run airflow initdb to create these missing tables.

This solved my issue.

Answer 2

The most common cause for this in airflow 1.10.12 is to add a $AIRFLOW_HOME/webserver_config.py file that overrides the default.

If you are adding a webserver_config.py it must contain

SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')

like in the default webserver_config.py.

In principle if you set rbac = True in airflow.cfg, remove webserver_config.py and execute any command like airflow list_users then airflow will create a webserver_config.py file for you with the proper SQLALCHEMY_DATABASE_URI = ... that you can modify afterwards.

Answer 3

You could patch app.py in airflow source code:

Set SQLALCHEMY_DATABASE_URI in app.config from SQL_ALCHEMY_CONN just before database initialization db = SQLA(app).

This is the path to app.py file:

/Users/deepaksaroha/Desktop/apache_2.0/nb-atom-airflow/lib/python3.7/site-packages/airflow/www_rbac/app.py

And this is the line you should include:

app.config['SQLALCHEMY_DATABASE_URI'] = conf.get('core', 'SQL_ALCHEMY_CONN')

Answer 4

The selected answer by @appleboy did not work for me, but the solution by @Nando_Quintana did.

Do the following to apply his fix as a patch

cd path/to/python/site-packages/airflow/www_rbac/
patch <<EOF
--- _app.py2019-09-25 17:18:59.127378849 +0000
+++ app.py 2019-09-25 17:18:53.467413221 +0000
@@ -53,6 +53,7 @@
     app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
     app.config['APP_NAME'] = app_name
     app.config['TESTING'] = testing
+    app.config['SQLALCHEMY_DATABASE_URI'] = conf.get('core', 'SQL_ALCHEMY_CONN')

     csrf.init_app(app)
EOF

Tested on Python 3.6.8 and Airflow 1.10.2

Answer 5

None of those worked for me. So figured out my own way. In the Postgres database mentioned in the config, drop the default "public" schema, and then create a new schema called "airflow" using your airflow postgres user. Then do

airflow db init

As I observed, the table ab_permission_view was being created in the public schema, whereas at a later stage the db init script was unable to locate it, as it looked for the table in the airflow schema mentioned in the config. Manually deleting the default schema and creating the required schema did the trick.