Use additional library in PySpark3 on HdInsight

Question

I am trying to install Shapely library but when I import it, I receive error message:

> from shapely.geometry import Point
No module named 'shapely'
Traceback (most recent call last):
ImportError: No module named 'shapely'

My kernel is :

cat /usr/local/share/jupyter/kernels/pyspark3kernel/kernel.json
{"argv":["python","-m","sparkmagic.kernels.pyspark3kernel.pyspark3kernel", "-f", "{connection_file}"],
 "display_name":"PySpark3"
}

I checked where my other libraries are installed:

> import pandas 
> pandas.__file__
'/usr/bin/anaconda/envs/py35/lib/python3.5/site-packages/pandas/__init__.py'

Shapely is installed in the same folder:

/usr/bin/anaconda/envs/py35/lib/python3.5/site-packages$ ls *hapely*
shapely:
affinity.py  _buildcfg.py  ctypes_declarations.py  examples   geometry   geos.py  __init__.py  linref.py  predicates.py  __pycache__  strtree.py   validation.py  wkb.py
algorithms   coords.py     errors.py               ftools.py  _geos.pxi  impl.py  iterops.py   ops.py     prepared.py    speedups     topology.py  vectorized     wkt.py

Shapely-1.6.3.dist-info:
DESCRIPTION.rst  INSTALLER  METADATA  metadata.json  RECORD  top_level.txt  WHEEL

FYI, my spark is over an HD Insight cluster.

What should I do or check to fix this library ? Let me know if you need more details.

EDIT:

I just found that

%%local
from shapely.geometry import Point

is working perfectly well... therefore, i think my lib is not installed on the cluster but is installed localy. How can I deploy it on the cluster ?

I don't understand, someone had to put them there. Was the cluster shipped from the vendor with them? — Paul H, Dec 15 '17 at 17:16
Basically the version installed is anaconda. Therefore it already comes with libs. That's the 1st time I try to install my own lib — Steven, Dec 15 '17 at 17:19
Yes. I did `pip`, I did `conda`. I can see my libs in local. But within spark, I dont see anythin. — Steven, Dec 15 '17 at 17:31
I'm not convinced that's the `py35` pandas. it might be the root environment's. can you copy/paste a terminal session where you activate the environment, launch terminal-based ipython, import pandas and pyspark, and show their paths? — Paul H, Dec 15 '17 at 17:51

score 1 · Accepted Answer · answered Dec 18 '17 at 14:59

As it is a spark cluster, lib must be installed on every node. Therefore, I need to execute the install command via the "script action" on the HD Insight interface. But, as I am using a virtual env, the command needs the -n env option to be added.

The final command to run is :

#!/usr/bin/env bash

/usr/bin/anaconda/bin/conda install -c conda-forge shapely -n py35

Use additional library in PySpark3 on HdInsight

1 Answers1