-3

I'm attempting to install pypandoc version 1.4 or higher. I included this requirement in my requirements.txt file. I believe that this is the reason why PySpark installation is failing.

my pip install command:

 python -m pip install -r requirements.txt --ignore-installed --exists-action=w --disable-pip-version-check --no-warn-script-location

the output:

enter image description here

amit
  • 71
  • 2
  • 10

2 Answers2

2

In your screenshot, you're installing pypandoc 1.11 (which is newer than 1.1 since version components are compared separately, and 11 > 1).

However, the issue here is that pypandoc hasn't been installed, it was just downloaded; if building pyspark requires it, it needs to have been separately installed first. (It should be declared a build dependency for pyspark, but pyspark 2.4.5 is 3 years old, so I doubt they're going to fix that).

In other words, first separately install pypandoc>1.4, then try again; at that point pypandoc should be importable by pyspark's setup.py.

If that still fails with the same symptom, you may need to add --no-build-isolation.

AKX
  • 152,115
  • 15
  • 115
  • 172
  • Thank you for your message. Due to the nature of Cloud Foundry, I am unable to separate these pip install commands as they need to be executed immediately after pushing the app. – amit May 03 '23 at 05:43
  • If you can run one arbitrary command, you can probably run two: `python -m pip install pypandoc>=1.4 && python -m pip install -r requirements.txt ...`. – AKX May 03 '23 at 05:56
  • I'm unable to do it myself. Once the push is done, the pip install command will run automatically. Is there a way to indicate in the requirements.txt file to install the initial package first, and then proceed with the rest? – amit May 03 '23 at 10:04
  • No, not as far as I know. – AKX May 03 '23 at 10:11
-1

Look in the requirements.txt file and see if what the pypandoc version it specifies in there. To ensure that pip installs the correct version of pypandoc, make sure your requirements.txt file contains the version constraint:

pypandoc>=1.4

If you still encounter issues, you can try installing pypandoc directly using the following command:

python -m pip install "pypandoc>=1.4"

If that works, you can try to install the other dependencies in your requirements.txt file using the original pip install command you provided.

If you still get the same problem you could try upgrading pip and setuptools using:

python -m pip install --upgrade pip setuptools

... then try installing the dependencies again.

FreddyC08
  • 51
  • 2