Why am I facing PermissionError while installing SpaCy en_core_web_sm on SSH server

Question

I am working on a project and have shifted my environment from local windows to a linux server (via SSH). I only have limited access as the host server is from my college, I've installed many packages without issues (both with and without virtualenv). I'm working on Python 3.6.9.

I was able to install spacy and import it but I need to use the en_core_web_sm package which has to be installed additionally using the command python3 -m spacy download en_core_web_sm. However, I consistently face a PermissionError as seen in the logs below.

Why am I facing this error? Is it because I don't have administrator access on the /usr level (refer to last line of logs)? If yes, how come only this package in particular requires a higher level access? If no, are there any workaround for me to install the package, or do I need to contact the server administrator?

jiayi@cdas1:~/adverse_drug_event_extraction/app/preprocessing$ python3 -m spacy download en_core_web_sm
Collecting en-core-web-sm==3.1.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.1.0/en_core_web_sm-3.1.0-py3-none-any.whl#egg=en_core_web_sm==3.1.0
  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.1.0/en_core_web_sm-3.1.0-py3-none-any.whl (13.6MB)
    100% |████████████████████████████████| 13.6MB 117kB/s 
Requirement already satisfied: spacy<3.2.0,>=3.1.0 in /home/jiayi/.local/lib/python3.6/site-packages (from en-core-web-sm==3.1.0)
Requirement already satisfied: pathy>=0.3.5 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: blis<0.8.0,>=0.4.0 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: typer<0.4.0,>=0.3.0 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: jinja2 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.7 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/lib/python3/dist-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: setuptools in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: catalogue<2.1.0,>=2.0.4 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: packaging>=20.0 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: typing-extensions<4.0.0.0,>=3.7.4; python_version < "3.8" in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: thinc<8.1.0,>=8.0.8 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: numpy>=1.15.0 in /home/jiayi/.local/lib/python3.6/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /home/jiayi/.local/lib/python3.6/site-packages (from pathy>=0.3.5->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: dataclasses<1.0,>=0.6; python_version < "3.7" in /home/jiayi/.local/lib/python3.6/site-packages (from pathy>=0.3.5->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: click<7.2.0,>=7.1.1 in /home/jiayi/.local/lib/python3.6/site-packages (from typer<0.4.0,>=0.3.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/jiayi/.local/lib/python3.6/site-packages (from jinja2->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: zipp>=0.5; python_version < "3.8" in /home/jiayi/.local/lib/python3.6/site-packages (from catalogue<2.1.0,>=2.0.4->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: pyparsing>=2.0.2 in /home/jiayi/.local/lib/python3.6/site-packages (from packaging>=20.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: contextvars<3,>=2.4; python_version < "3.7" in /home/jiayi/.local/lib/python3.6/site-packages (from thinc<8.1.0,>=8.0.8->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Requirement already satisfied: immutables>=0.9 in /home/jiayi/.local/lib/python3.6/site-packages (from contextvars<3,>=2.4; python_version < "3.7"->thinc<8.1.0,>=8.0.8->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0)
Installing collected packages: en-core-web-sm
Exception:
Traceback (most recent call last):
  File "/home/jiayi/.local/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/jiayi/.local/lib/python3.6/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/home/jiayi/.local/lib/python3.6/site-packages/pip/req/req_set.py", line 784, in install
    **kwargs
  File "/home/jiayi/.local/lib/python3.6/site-packages/pip/req/req_install.py", line 851, in install
    self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
  File "/home/jiayi/.local/lib/python3.6/site-packages/pip/req/req_install.py", line 1064, in move_wheel_files
    isolated=self.isolated,
  File "/home/jiayi/.local/lib/python3.6/site-packages/pip/wheel.py", line 345, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/home/jiayi/.local/lib/python3.6/site-packages/pip/wheel.py", line 316, in clobber
    ensure_dir(destdir)
  File "/home/jiayi/.local/lib/python3.6/site-packages/pip/utils/__init__.py", line 83, in ensure_dir
    os.makedirs(path)
  File "/usr/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/en_core_web_sm-3.1.0.dist-info'

polm23 · Accepted Answer · 2021-08-26T05:02:29.420

As you suspect, this is happening because you don't have permission to install to /usr. Normally pip would install there, but it looks like your pip is basically running with --user, which will install to your user home directory instead.

The spaCy models can't be installed directly via pip because they are large data files, so they can't be hosted on PyPI like ordinary code. An unfortunate side effect of this is that some of the options around pip configuration are ignored.

You can pass extra arguments to the pip install command by appending them to your command, so in your case you can do this:

python3 -m spacy download en_core_web_sm --user

And everything should work.

That said, I would strongly recommend you learn how to use virtual environments with the virtualenv tool, which will make working with Python projects easier and allow to you avoid this problem, as anything you install will just go in the local virtualenv rather than your global pip install.

score 1 · Answer 2 · answered Aug 25 '21 at 08:45

It seems all other packages are installed under your user /home/jiayi/.local/python3.6/lib and this one tries to install itself globally in /usr/local/lib/python3.6/, not sure why. I guess you can give it installation folder or something. Look here Where does spacy language model download?

Why am I facing PermissionError while installing SpaCy en_core_web_sm on SSH server

2 Answers2