How to build and install pylucene on ubuntu 20.04

Question

I am trying to install Pylucene on my WSL Ubuntu 20.04 clean installation. I tried to follow tutorial on the official page but it looks outdated. So I was wondering if anyone here managed to make it work on Ubuntu 20.04 and python 3.8.2

The commands I run:

sudo apt-get upgrade
sudo apt-get install -y default-jdk ant build-essential python3-dev
mkdir pylucene
cd pylucene
curl https://downloads.apache.org/lucene/pylucene/pylucene-8.3.0-src.tar.gz | tar -xz --strip-components=1
cd jcc
export JCC_JDK=/usr/lib/jvm/default-java
python3 setup.py build

^^^^^ Fails here on:

...
building 'jcc3' extension
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -D_jcc_lib -DJCC_VER="3.7" -I/usr/lib/jvm/default-java/include -I/usr/lib/jvm/default-java/include/linux -I_jcc3 -Ijcc3/sources -I/usr/include/python3.8 -c jcc3/sources/jcc.cpp -o build/temp.linux-x86_64-3.8/jcc3/sources/jcc.o -DPYTHON -fno-strict-aliasing -Wno-write-strings
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -D_jcc_lib -DJCC_VER="3.7" -I/usr/lib/jvm/default-java/include -I/usr/lib/jvm/default-java/include/linux -I_jcc3 -Ijcc3/sources -I/usr/include/python3.8 -c jcc3/sources/JCCEnv.cpp -o build/temp.linux-x86_64-3.8/jcc3/sources/JCCEnv.o -DPYTHON -fno-strict-aliasing -Wno-write-strings
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/jcc3/sources/jcc.o build/temp.linux-x86_64-3.8/jcc3/sources/JCCEnv.o -o build/lib.linux-x86_64-3.8/libjcc3.so -L/usr/lib/jvm/default-java/jre/lib/amd64 -ljava -L/usr/lib/jvm/default-java/jre/lib/amd64/server -ljvm -Wl,-rpath=/usr/lib/jvm/default-java/jre/lib/amd64:/usr/lib/jvm/default-java/jre/lib/amd64/server -Wl,-S
/usr/bin/ld: cannot find -ljava
/usr/bin/ld: cannot find -ljvm
collect2: error: ld returned 1 exit status
error: command 'x86_64-linux-gnu-g++' failed with exit status 1

Commands I plan to run afterwards:

sudo python3 --preserve-env=JCC_JDK setup.py install
cd ..
make
make test
sudo make install

Have you checked if `libjava.so` and `libjvm.so` exist in any of the paths mentioned to the `-L` options or anywhere else in your system? — bracco23, Aug 17 '20 at 14:34
I would suggest going into your `/usr/lib/jvm` folder and performing a search to see if you can find the file anywhere (a simple `find . -name "libjvm.so"` should be enough). After that, once you know where they are, you can find what additional env var you should set to have the setup correctly find them too (i suspect `JCC_JDK` is pointing to the wrong directory but i cannot prove it) — bracco23, Aug 17 '20 at 14:43

Code Your Dream · Answer 1 · 2021-01-27T02:00:49.147

Here are the steps that lead to the successful installation of pylucene on Ubuntu 18.04 - this may work for you:

Install openjdk-8: apt install openjdk-8-jre openjdk-8-jdk openjdk-8-doc Ensure that you have ant installed, if you don't run apt install ant. Note that if you had a different version of openjdk installed you need to either remove it or run update-alternatives so that version 1.8.0 is used.
Check that Java version is 1.8.0* with java -version
After installing openjdk-8 create a symlink (you'll need it later):

cd /usr/lib/jvm
ln -s java-8-openjdk-amd64 java-8-oracle

Install python-dev: sudo apt install python-dev In my case Python 3 didn't work so I ended up using Python 2. But this might not have been the actual reason of the problem, so you're welcome to try Python 3. If you go with Python 3, use python3 instead of python in the commands below.
Install JCC (in jcc subfolder of your pylucene folder):

python setup.py build
python setup.py install

The symlink you created on step 3 will help here because this path is hardcoded into setup.py - you can check that.

Install pylucene (from the root of your pylucene folder). Edit Makefile, uncomment/edit the variables according to your setup. In my case it was

PREFIX_PYTHON=/usr
ANT=ant
PYTHON=$(PREFIX_PYTHON)/bin/python
JCC=$(PYTHON) -m jcc --shared
NUM_FILES=10

Then run

make
make test
sudo make install

If you see an error related to the shared mode of JCC remove --shared from Makefile.

score 0 · Answer 2 · answered Jan 25 '21 at 17:11

Confirmed that the answer posted by @code-your-dream works also with Python3 (and in particular in a Ubuntu 18.04.1)

For me was important to install jcc that way. I tried installing via conda and there were conflicts in the make of pylucene.

In my case also was needed to modify the setup.py file from jcc changing 'linux': ['-fno-strict-aliasing', '-Wno-write-strings'], by 'linux': ['-fno-strict-aliasing', '-Wno-write-strings','-D__STDC_FORMAT_MACROS'], as mentioned in this other thread Issue with installing PyLucene 6.5.0 on Linux

To confirm which is the version of java needed (in my case also 8, didn't work with 11), you can search in the pylucene folder /jcc/setup.py the block JDK = {...}. In my case with the reference: 'linux': '/usr/lib/jvm/java-8-oracle',

score 0 · Answer 3 · answered Aug 22 '22 at 09:05

I wrote this dockerfile for PyLucene 8.11.0, Python 3.9, JDK11 (default-jdk) and ubuntu 20.04 (focal). Do extend with your favorite python package manager such as Poetry, Pipenv or Conda etc.

Dockerfile

FROM ubuntu:focal

ARG PYTHON_VERSION=3.9
ARG PYLUCENE_VERSION=8.11.0

# Uncomment to install specific version of poetry
ENV LANG=C.UTF-8

# ADD Python PPA Repository
RUN apt-get update && \
    apt-get upgrade -y && \
    apt-get install -y --no-install-recommends \
        software-properties-common gpg-agent && \
    add-apt-repository ppa:deadsnakes/ppa && \
    apt-get update && \
    apt-get remove -y software-properties-common && \
    apt-get purge --auto-remove -y && \
    apt-get clean

RUN which gpg-agent

# Install Python
RUN apt-get install -y --no-install-recommends \
        "python$PYTHON_VERSION-dev" \
        python3-setuptools \
        python3-pip && \
    apt-get remove -y gpg-agent && \
    apt-get purge --auto-remove -y && \
    apt-get clean

# ======================== START OF ADDITIONAL INSTALLATION ========================

# Install Java
RUN apt-get install -y --no-install-recommends \
    build-essential \
    ant \
    jcc \
    curl \
    git \
    default-jdk 
    
RUN ls /usr/bin/ | grep "python"
RUN ln -s $(which python3.9) /usr/bin/python

RUN which python3.9 && which python && python --version

WORKDIR /usr/lib/jvm/default-java/jre/lib
RUN ln -s ../../lib amd64

# Java 11
RUN java --version && javac --version


# Installing PyLucene
RUN which ant && ant -version

RUN apt-get install -y --no-install-recommends \
    libffi-dev \
    zlib1g-dev

WORKDIR /usr/src/pylucene
RUN curl https://dlcdn.apache.org/lucene/pylucene/pylucene-$PYLUCENE_VERSION-src.tar.gz | tar -xz


ENV PREFIX_PYTHON=/usr \
    JCC_JDK=/usr/lib/jvm/default-java \
    ANT=ant \
    JCC='python -m jcc' \
    NUM_FILES=10 \
    PYTHON=python \
    NO_SHARED=1

RUN cd "pylucene-$PYLUCENE_VERSION/lucene-java-$PYLUCENE_VERSION/lucene" && \
    ant ivy-bootstrap && \
    ant && \
    cd ../../../

RUN cd "pylucene-$PYLUCENE_VERSION/jcc" && \
    ls -la && \
    NO_SHARED=1 JCC_JDK=/usr/lib/jvm/default-java python setup.py build && \
    NO_SHARED=1 JCC_JDK=/usr/lib/jvm/default-java python setup.py install && \
    cd .. && \
    make JCC="python -m jcc" ANT=ant PYTHON=python NUM_FILES=8&& \
    make install JCC="python -m jcc" ANT=ant PYTHON=python NUM_FILES=8 && \
    cd ../../

RUN apt-get remove -y gpg-agent ant jcc build-essential && \
    apt-get purge --auto-remove -y && \
    apt-get clean

WORKDIR /usr/src
RUN rm -rf pylucene

RUN python -c "import lucene; lucene.initVM()"

# ======================== END OF ADDITIONAL INSTALLATION ========================

WORKDIR /app
COPY . .

How to build and install pylucene on ubuntu 20.04

3 Answers3