0

We are using JDBC Connection in airflow to fetch data from a public MySQL server. The connection was setup fine and worked as expected in the local machine but I received the following error in composer.

[Errno 2] No such file or directory: '/usr/lib/jvm'

I m currently thinking if

  • Am I doing something wrong ?
  • JVM is unavailable in Composer webserver?
Tameem
  • 408
  • 7
  • 19
  • JVM is unavailable in Composer Webserver. Airflow runs on python, I do not believe any Java is executed in the stack. If it is a public MySQL server is there any reason you can't just use the MySQL operator? If there is try using an implementation similar to the MySQL Hook, as these packages will exist: https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/mysql_hook.py – HulaHoof Nov 12 '18 at 00:44
  • 1
    Airflow is built on python. But, the Jdbc Connections use JayDeBeApi. Which provides an ability to connect to servers using JDBC. Thus java IS executed on stack; just as bash is during BashOperator. For more info you can check out the source code at https://github.com/apache/incubator-airflow/blob/master/airflow/operators/jdbc_operator.py – Tameem Nov 12 '18 at 04:16
  • Ah you are correct. Interestingly I could not import jaydebeapi from my workers. The /usr/lib/jvm directory is there however (as you are probably aware). Can you run it over CLI via gcloud or within a worker? – HulaHoof Nov 12 '18 at 04:39
  • To use jaydebeapi you'll have to install it as a dependency package. The DAGs in our case are dynamically generated via entries in the database.We are able to run the operator through worker but as the webserver does not have JVM the part where we connect to MySQL fails and DAGs do not show up in the webserver. – Tameem Nov 12 '18 at 06:10

0 Answers0