external library error while running AWS Glue job

Asked Sep 10 '20 at 22:04

Active Sep 10 '20 at 22:17

Viewed 354 times

I have placed the external python libraries (*.whl) in S3 and is accessing the same by mentioning the path in AWS Glue Job 'Python library path' argument.

It runs fine for few external modules but fails for others with the below error:

Traceback (most recent call last):
  File "/tmp/processing", line 2, in <module> import netCDF4 File "/tmp/netCDF4-1.5.4-cp36-cp36m-manylinux1_x86_64.whl/netCDF4/__init__.py", line 3, in <module>
    from ._netCDF4 import *
ModuleNotFoundError: No module named 'netCDF4._netCDF4'

Any suggestions will be appreciated.

edited Sep 10 '20 at 22:17

anthony sottile

61,815
15
148
207

asked Sep 10 '20 at 22:04

Sandip Mondal

using wheels on `sys.path` doesn't work for binary dependencies (`*.so`) – anthony sottile Sep 10 '20 at 22:17
What shall I do to solve it or any workaround? – Sandip Mondal Sep 10 '20 at 22:30

external library error while running AWS Glue job

0 Answers0