lxml will never finish building on ubuntu

Question

I am running ubuntu 14.04 LTS and python 2.7.5 on a vwmare.

When I run:

sudo pip install lxml

I get:

Collecting lxml
  Using cached lxml-3.4.4.tar.gz
Building wheels for collected packages: lxml
  Running setup.py bdist_wheel for lxml

which runs eternally.

I can't find any resources related to this, nor do I have any experience with similar errors.

Note, the solution below does not work:

sudo apt-get build-dep -y lxml;
pip install lxml

I have also tried:

sudo apt-get install -y build-essential python-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
build-essential is already the newest version.
python-dev is already the newest version.
The following packages were automatically installed and are no longer required:
  linux-headers-3.13.0-32 linux-headers-3.13.0-32-generic
  linux-headers-3.13.0-57 linux-headers-3.13.0-57-generic
  linux-image-3.13.0-32-generic linux-image-3.13.0-57-generic
  linux-image-extra-3.13.0-32-generic linux-image-extra-3.13.0-57-generic
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 75 not upgraded.

EDIT:

I just ran it again and got:

sudo pip install lxml

Collecting lxml
/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached lxml-3.4.4.tar.gz
Building wheels for collected packages: lxml
  Running setup.py bdist_wheel for lxml

EDIT 2:

 pip install lxml

The directory '/home/myusername/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/myusername/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting lxml
/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading lxml-3.4.4.tar.gz (3.5MB)
    100% |████████████████████████████████| 3.5MB 7.0kB/s 
Installing collected packages: lxml
  Running setup.py install for lxml

EDIT 3: After waiting a long time the command gave:

pip install lxml
The directory '/home/mstaffel/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/mstaffel/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting lxml
/usr/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading lxml-3.4.4.tar.gz (3.5MB)
    100% |████████████████████████████████| 3.5MB 7.0kB/s 
Installing collected packages: lxml
  Running setup.py install for lxml
    Complete output from command /usr/local/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-u3iJ67/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-0tAWPp-record/install-record.txt --single-version-externally-managed --compile:
    /usr/local/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
      warnings.warn(msg)
    Building lxml version 3.4.4.
    Building without Cython.
    Using build configuration of libxslt 1.1.28
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-i686-2.7
    creating build/lib.linux-i686-2.7/lxml
    copying src/lxml/pyclasslookup.py -> build/lib.linux-i686-2.7/lxml
    copying src/lxml/cssselect.py -> build/lib.linux-i686-2.7/lxml
    copying src/lxml/__init__.py -> build/lib.linux-i686-2.7/lxml
    copying src/lxml/_elementpath.py -> build/lib.linux-i686-2.7/lxml
    copying src/lxml/doctestcompare.py -> build/lib.linux-i686-2.7/lxml
    copying src/lxml/builder.py -> build/lib.linux-i686-2.7/lxml
    copying src/lxml/ElementInclude.py -> build/lib.linux-i686-2.7/lxml
    copying src/lxml/usedoctest.py -> build/lib.linux-i686-2.7/lxml
    copying src/lxml/sax.py -> build/lib.linux-i686-2.7/lxml
    creating build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/__init__.py -> build/lib.linux-i686-2.7/lxml/includes
    creating build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/__init__.py -> build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/_diffcommand.py -> build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/soupparser.py -> build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/builder.py -> build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/usedoctest.py -> build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/html5parser.py -> build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/_setmixin.py -> build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/_html5builder.py -> build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/defs.py -> build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/clean.py -> build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/ElementSoup.py -> build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/diff.py -> build/lib.linux-i686-2.7/lxml/html
    copying src/lxml/html/formfill.py -> build/lib.linux-i686-2.7/lxml/html
    creating build/lib.linux-i686-2.7/lxml/isoschematron
    copying src/lxml/isoschematron/__init__.py -> build/lib.linux-i686-2.7/lxml/isoschematron
    copying src/lxml/lxml.etree.h -> build/lib.linux-i686-2.7/lxml
    copying src/lxml/lxml.etree_api.h -> build/lib.linux-i686-2.7/lxml
    copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/tree.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/schematron.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/xinclude.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/c14n.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/xpath.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/xslt.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/uri.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/relaxng.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/config.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/etree_defs.h -> build/lib.linux-i686-2.7/lxml/includes
    copying src/lxml/includes/lxml-version.h -> build/lib.linux-i686-2.7/lxml/includes
    creating build/lib.linux-i686-2.7/lxml/isoschematron/resources
    creating build/lib.linux-i686-2.7/lxml/isoschematron/resources/rng
    copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-i686-2.7/lxml/isoschematron/resources/rng
    creating build/lib.linux-i686-2.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-i686-2.7/lxml/isoschematron/resources/xsl
    copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-i686-2.7/lxml/isoschematron/resources/xsl
    creating build/lib.linux-i686-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-i686-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-i686-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-i686-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-i686-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-i686-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-i686-2.7/lxml/isoschematron/resources/xsl/iso-schematron-xslt1
    running build_ext
    building 'lxml.etree' extension
    creating build/temp.linux-i686-2.7
    creating build/temp.linux-i686-2.7/src
    creating build/temp.linux-i686-2.7/src/lxml
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include/libxml2 -I/tmp/pip-build-u3iJ67/lxml/src/lxml/includes -I/usr/local/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-i686-2.7/src/lxml/lxml.etree.o -w
    gcc -pthread -shared build/temp.linux-i686-2.7/src/lxml/lxml.etree.o -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-i686-2.7/lxml/etree.so
    building 'lxml.objectify' extension
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/include/libxml2 -I/tmp/pip-build-u3iJ67/lxml/src/lxml/includes -I/usr/local/include/python2.7 -c src/lxml/lxml.objectify.c -o build/temp.linux-i686-2.7/src/lxml/lxml.objectify.o -w
    gcc -pthread -shared build/temp.linux-i686-2.7/src/lxml/lxml.objectify.o -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-i686-2.7/lxml/objectify.so
    running install_lib
    creating /usr/local/lib/python2.7/site-packages/lxml
    error: could not create '/usr/local/lib/python2.7/site-packages/lxml': Permission denied

    ----------------------------------------
Command "/usr/local/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-u3iJ67/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-0tAWPp-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-u3iJ67/lxml

That also did not work, I have added its result to the question. @BurhanKhalid — Rorschach, Oct 11 '15 at 11:33
Thank you for helping though, if you've got any other ideas I would really appreciate it — Rorschach, Oct 11 '15 at 11:33
`sudo apt-get install -y libxml2-dev libxslt-dev zlib1g-dev`, then give me the exact output for `pip install lxml` - note that if you are running this as a normal user, you should get file system permission errors (unless you were running it inside a virtual environment). — Burhan Khalid, Oct 11 '15 at 11:41
I will add it to the question, since it will not fit here, it is under Edit2 @BurhanKhalid — Rorschach, Oct 11 '15 at 11:45
It just recently gave more output, it is under edit 3 @BurhanKhalid — Rorschach, Oct 11 '15 at 12:17
`sudo pip install lxml`; since you are installing it across the system. — Burhan Khalid, Oct 11 '15 at 12:41
I am running it now, and waiting to see if it ever finishes, last time it took ~30 mins — Rorschach, Oct 11 '15 at 12:45

amorphic · Accepted Answer · 2018-07-17T11:39:12.603

22

Note that if you run top in a separate terminal you'll notice that your system is busy building the lxml package:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1645 root 20 0 322224 305740 5384 R 100.0 29.8 3:27.47 /usr/lib/gcc/arm-linux-gnueabihf/5/cc1 -quiet -I /usr/include/libxml2 -I src/lxml/includes -I /usr/include/python2.7 -imultiarch arm-linux-gnueabihf -D_REENTRANT -D ...

If you're building on a VM or SBC with limited resources this can take a while...

edited Jul 17 '18 at 11:39

answered Mar 12 '17 at 11:18

amorphic

722
8
12

1

You're a king. This process took like 10 minutes for me to complete. I was going insane. – David T Jul 16 '18 at 11:31

score 6 · Answer 2 · answered Sep 09 '19 at 12:11

I ran into the same problem when trying to "pip install" a package that depends on lxml on a Raspberry Pi 1 B+. After several hours (and several attempts), the SSH connection dropped without finishing the wheel build. It seems like an out-of-memory situation.

I opted on installing the pre-built binary packages using apt install python-lxml or apt install python3-lxml respectively.

score 4 · Answer 3 · answered Nov 15 '18 at 12:44

4

I ran this on VM with 768MB ram (which i didn't notice was the value i set) and it took several hours and didn't finish.

Changing ram to 3GB made it finish in few minutes after restarting and trying again.

answered Nov 15 '18 at 12:44

Riki137

2,076
2
23
26

Pranav Kumar · Answer 4 · 2021-07-28T17:07:50.027

2

The newest version of pip, unnecessarily spends excessive amount of time to build wheels and resolve dependency issues while building a package with requirements.txt or a setuptools/distutils setup.py

A quick fix is to revert to a version of pip prior to 21.0 (ideally in a virtual environment), where the legacy resolver is still supported, according to this discussion: sunsetting legacy resolver

As of July 28th, 2021 pip v21.2.1 still seems to have the legacy resolver support.

pip install pip==20.2.4

Command below uses an earlier version of dependency resolver to install lxml.

pip install --use-deprecated=legacy-resolver lxml

Saved me a lot of time and hassle.

edited Jul 28 '21 at 17:07

answered Jul 26 '21 at 20:03

Pranav Kumar

254
2
7

Doesn't work for me: (pip `20.2.4`): "option --use-deprecated: invalid choice: 'legacy-resolver' (choose from )" – TeNNoX Mar 15 '22 at 10:27
Ah, but it works with `pip 21.2.4` :thumbsup: - maybe you meant to write that in the first code block ? – TeNNoX Mar 15 '22 at 10:28

score 0 · Answer 5 · answered May 24 '21 at 16:28

0

I think you should to install cython. As I can see in your code it is said that: Building without Cython. and if we go to pypi and look up for lxml it is said that Note that this requires Cython to build the sources, see the build instructions on the project home page. Here is the link: lxml

answered May 24 '21 at 16:28

d9nam1c

1
1

That's only for the development version, for any stable version you don't need Cython and they recommend against it: "Since we distribute the Cython-generated .c files with lxml releases, however, you do not need Cython to build lxml from the normal release sources. We even encourage you to not install Cython for a normal release build, as the generated C code can vary quite heavily between Cython versions, which may or may not generate correct code for lxml." (https://lxml.de/build.html) – Lewis Aug 08 '23 at 13:02

lxml will never finish building on ubuntu

5 Answers5