I have custom libraries that I have installed on my cluster using EGG files and also on my local machine. I use databricks connect to connect my IDE to my cluster. When I try importing the module, it gives me error module not found when executed using IDE. The same code runs perfectly when executed using databricks notebooks. I also tried to install public modules like httpagentparser onto the cluster and my local machine still these installed modules are not found while executed on IDE through databricks connect but runs perfectly on notebooks. I have httpagentparser installed locally and it runs fine when tested individually.
Asked
Active
Viewed 358 times
3
-
are you sure that IDE uses the same python environment where databricks-connect is installed together with your custom library? – Alex Ott Jul 03 '21 at 08:32
-
Yes, I can use all the libraries present default like Math class on the remote cluster. I also make sure that the command is executed on the remote clusters as I can see them in the spark UI. – Sagar Moghe Jul 04 '21 at 14:01