0

I'm trying to use turbodbc with Pyarrow support on Databricks - runtime 7.2. Turbodbc works without the pyarrow support well on the same same instance. From Databricks 7.2 release page it says that Pyarrow is already which I've verified to be true. It comes with 0.15.1.

However, when i try running turbodbc with pyarrow, I'm thrown the below error.

This installation of turbodbc does not support Apache Arrow extensions. Please install the pyarrow package. If you have built turbodbc from source, you may also need to reinstall turbodbc to compile the extensions.

The databricks is bases on Ubuntu 18.04. I've made sure that the init script installs

libboost-all-dev 
unixodbc-dev 
python-dev

I also tried specifying pyarrow again in the libraries list so that it gets installed via pip, but no luck.

Any help is greatly appreciated.

Update 1:

Tried creating an ML cluster where python package installation happens with conda. I'm still getting the same error.

Vijay
  • 1,030
  • 11
  • 34
  • Regarding "Tried creating an ML cluster where python package installation happens with conda. I'm still getting the same error." Did you install turbodbc via conda? That should fix your issue. – Uwe L. Korn Sep 10 '20 at 08:42
  • Yes I did! However, conda installs the version 3.0.0 of turbodbc. When I try running a sql query using this version and reading into a Pyarrow table, I get a connection closed error. Also I checked numpy, pyarrow versions and they seem to fit the requirement for latest version of turbodbc. I'm on Ubuntu 18.04 – Vijay Sep 10 '20 at 09:04
  • What happens when you do `conda install turbodbc=4.1.1`? – Uwe L. Korn Sep 11 '20 at 17:57
  • I don't exactly remember the message, but it keeps to trying to get the package suitable for the environment. I waited for almost 20+ min and nothing happened. – Vijay Sep 12 '20 at 17:00
  • That sounds like a conflict with another installed package. You could try using `mamba install` as an alternative to the `conda` command: https://github.com/mamba-org/mamba This should give you a better error message. – Uwe L. Korn Sep 13 '20 at 19:30
  • 1
    I don't think mamba is available on Databricks. Let me see if I can install mamba and try installing. – Vijay Sep 14 '20 at 17:43

0 Answers0