ModuleNotFoundError: No module named 'numpy.random._pickle'

Question

I have a doc2vec model which drives my recommendation app. I have built the doc2vec model and saved into s3 bucket. Now when i open the webapp the model should be loaded back from s3 but this not happenning. I used AWS Elasticbean stalk to deploy my webapplication. So, i then checked the logs of Elasticbean stalk for errors. This is the error it shows me when i try to deploy

-------------------------------------
/var/log/httpd/error_log
-------------------------------------
[Fri May 22 12:53:57.768095 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/usr/lib64/python3.6/pickle.py", line 1338, in load_global
[Fri May 22 12:53:57.768098 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     klass = self.find_class(module, name)
[Fri May 22 12:53:57.768104 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/usr/lib64/python3.6/pickle.py", line 1388, in find_class
[Fri May 22 12:53:57.768107 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     __import__(module, level=0)
[Fri May 22 12:53:57.768127 2020] [:error] [pid 8177] [remote 172.31.23.26:48093] ModuleNotFoundError: No module named 'numpy.random._pickle'
[Fri May 22 12:53:57.768138 2020] [:error] [pid 8177] [remote 172.31.23.26:48093] 
[Fri May 22 12:53:57.768141 2020] [:error] [pid 8177] [remote 172.31.23.26:48093] During handling of the above exception, another exception occurred:
[Fri May 22 12:53:57.768144 2020] [:error] [pid 8177] [remote 172.31.23.26:48093] 
[Fri May 22 12:53:57.768150 2020] [:error] [pid 8177] [remote 172.31.23.26:48093] Traceback (most recent call last):
[Fri May 22 12:53:57.768177 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/opt/python/current/app/application.py", line 81, in <module>
[Fri May 22 12:53:57.768181 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     model_d2v = d2v.load_d2v('doc2vec_model', ENV)
[Fri May 22 12:53:57.768187 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/opt/python/current/app/microservices/d2v_services.py", line 21, in load_d2v
[Fri May 22 12:53:57.768190 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     model=joblib.load(model_name)
[Fri May 22 12:53:57.768196 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 605, in load
[Fri May 22 12:53:57.768199 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     obj = _unpickle(fobj, filename, mmap_mode)
[Fri May 22 12:53:57.768204 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 529, in _unpickle
[Fri May 22 12:53:57.768208 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     obj = unpickler.load()
[Fri May 22 12:53:57.768213 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/usr/lib64/python3.6/pickle.py", line 1050, in load
[Fri May 22 12:53:57.768216 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     dispatch[key[0]](self)
[Fri May 22 12:53:57.768222 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/usr/lib64/python3.6/pickle.py", line 1338, in load_global
[Fri May 22 12:53:57.768230 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     klass = self.find_class(module, name)
[Fri May 22 12:53:57.768237 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]   File "/usr/lib64/python3.6/pickle.py", line 1388, in find_class
[Fri May 22 12:53:57.768240 2020] [:error] [pid 8177] [remote 172.31.23.26:48093]     __import__(module, level=0)
[Fri May 22 12:53:57.768250 2020] [:error] [pid 8177] [remote 172.31.23.26:48093] ModuleNotFoundError: No module named 'numpy.random._pickle'
[Fri May 22 12:54:07.192783 2020] [:error] [pid 8177] loading...doc2vec_model
[Fri May 22 12:54:12.237091 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] mod_wsgi (pid=8177): Target WSGI script '/opt/python/current/app/application.py' cannot be loaded as Python module.
[Fri May 22 12:54:12.237334 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] mod_wsgi (pid=8177): Exception occurred processing WSGI script '/opt/python/current/app/application.py'.
[Fri May 22 12:54:12.238604 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] Traceback (most recent call last):
[Fri May 22 12:54:12.238658 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/current/app/microservices/d2v_services.py", line 14, in load_d2v
[Fri May 22 12:54:12.238664 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     model=joblib.load(model_name)
[Fri May 22 12:54:12.238675 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 605, in load
[Fri May 22 12:54:12.238679 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     obj = _unpickle(fobj, filename, mmap_mode)
[Fri May 22 12:54:12.238686 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 529, in _unpickle
[Fri May 22 12:54:12.238690 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     obj = unpickler.load()
[Fri May 22 12:54:12.238695 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/usr/lib64/python3.6/pickle.py", line 1050, in load
[Fri May 22 12:54:12.238699 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     dispatch[key[0]](self)
[Fri May 22 12:54:12.238705 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/usr/lib64/python3.6/pickle.py", line 1338, in load_global
[Fri May 22 12:54:12.238709 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     klass = self.find_class(module, name)
[Fri May 22 12:54:12.238714 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/usr/lib64/python3.6/pickle.py", line 1388, in find_class
[Fri May 22 12:54:12.238718 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     __import__(module, level=0)
[Fri May 22 12:54:12.238737 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] ModuleNotFoundError: No module named 'numpy.random._pickle'
[Fri May 22 12:54:12.238793 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] 
[Fri May 22 12:54:12.238800 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] During handling of the above exception, another exception occurred:
[Fri May 22 12:54:12.238803 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] 
[Fri May 22 12:54:12.238810 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] Traceback (most recent call last):
[Fri May 22 12:54:12.238840 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/current/app/application.py", line 81, in <module>
[Fri May 22 12:54:12.238845 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     model_d2v = d2v.load_d2v('doc2vec_model', ENV)
[Fri May 22 12:54:12.238851 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/current/app/microservices/d2v_services.py", line 21, in load_d2v
[Fri May 22 12:54:12.238855 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     model=joblib.load(model_name)
[Fri May 22 12:54:12.238872 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 605, in load
[Fri May 22 12:54:12.238883 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     obj = _unpickle(fobj, filename, mmap_mode)
[Fri May 22 12:54:12.238889 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/opt/python/run/venv/local/lib/python3.6/site-packages/joblib/numpy_pickle.py", line 529, in _unpickle
[Fri May 22 12:54:12.238892 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     obj = unpickler.load()
[Fri May 22 12:54:12.238897 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/usr/lib64/python3.6/pickle.py", line 1050, in load
[Fri May 22 12:54:12.238901 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     dispatch[key[0]](self)
[Fri May 22 12:54:12.238906 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/usr/lib64/python3.6/pickle.py", line 1338, in load_global
[Fri May 22 12:54:12.238909 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     klass = self.find_class(module, name)
[Fri May 22 12:54:12.238914 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]   File "/usr/lib64/python3.6/pickle.py", line 1388, in find_class
[Fri May 22 12:54:12.238917 2020] [:error] [pid 8177] [remote 172.31.15.191:48093]     __import__(module, level=0)
[Fri May 22 12:54:12.238928 2020] [:error] [pid 8177] [remote 172.31.15.191:48093] ModuleNotFoundError: No module named 'numpy.random._pickle'
[Fri May 22 12:56:22.918445 2020] [mpm_prefork:notice] [pid 8172] AH00169: caught SIGTERM, shutting down

I tried importing 'numpy.random._pickle' but it didn't work.

The code I used for loading the model:

import subprocess
from sklearn.externals import joblib
import gensim

model = load_d2v("doc2vec_model")

def load_d2v(fname):
    model_name = fname
    s3_base_path='s3://sd-flikku/datalake/current_doc2vec_model'
    path = s3_base_path+'/'+model_name  
    command = "aws s3 cp {} {}".format(path,model_name).split()
    print('loading...'+model_name)
    subprocess.call(command)
    model=joblib.load(model_name)
    return model

Can you please help me with this? Thanks in advance.

Is `numpy` installed where this code is running, and the same version as where the model was originally saved? — gojomo, May 26 '20 at 15:54

score 4 · Answer 1 · answered Jun 01 '20 at 09:28

4

Upgrade the numpy library using:

pip install numpy --upgrade

Occurs when the model was built with a specific version of the library and tested with other versions.

answered Jun 01 '20 at 09:28

Santosh Gunashekar

71
1

score 3 · Accepted Answer · answered May 26 '20 at 07:27

3

Most likely this is numpy version mismatch.

This can come from old versions that are not completely uninstalled or wrong version to the packages you're using.

These are people with the same problem

answered May 26 '20 at 07:27

Tom Nijhof

542
4
11

ModuleNotFoundError: No module named 'numpy.random._pickle'

2 Answers2