Using CGI - python having problem finding modules

Question

I am trying to run a python-script via CGI but I get Internal server error

error log according to apache2:

[Mon Dec 21 10:43:19.073771 2020] [cgi:error] [pid 7531] [client ::1:50038] AH01215:   File  "/usr/lib/cgi-bin/pytest.py", line 5, in <module>: /usr/lib/cgi-bin/pytest.py
[Mon Dec 21 10:43:19.073809 2020] [cgi:error] [pid 7531] [client ::1:50038] AH01215:     from  bs4 import BeautifulSoup                                       : /usr/lib/cgi-bin/pytest.py
[Mon Dec 21 10:43:19.073832 2020] [cgi:error] [pid 7531] [client ::1:50038] AH01215:  ModuleNotFoundError: No module named 'bs4': /usr/lib/cgi-bin/pytest.py

It's all about a pretty simple scraper-script that I want to run via CGI. I would emphasize that running this script normally through terminal is no problem, its just a problem using CGI

Is it possible to solve this, or is CGI not fitted for these kind of scripts?

code:

#! /usr/bin/python3

# enable debugging
import cgitb
from bs4 import BeautifulSoup                                       
from urllib.request import urlopen, Request
import os

cgitb.enable()

print ("Content-type: text/html\r\n\r\n")

print("hello")


url = "https://www.xxxxxxxxxxxxx"

soup = BeautifulSoup(urlopen(url).read())

...

Your shebang looks wrong. How does it run at all? When you run it from the command line are you using the same Python executable? Or is the PYTHONPATH environment variable set up? Or are you running out of a virtual environment? — Booboo, Dec 21 '20 at 11:07
@Booboo - sorry, just a typo here when pasting the code into stackoverflow! Well it's not the same executable but the same code. As I wrote I can run the script in the terminal ( ./scraper.py) on Debian but not through the url. I can also add that I can actually run this script through cgi, IF i comment out them import command and the rest of the beautifulsoup syntax of course. I can at the least print hello on the page using python3 I'll edit the text — java, Dec 21 '20 at 11:25
I thought it might be a typo. If it is not the same Python executable, that's a problem, You are therefore running an installation of Python that doesn't have BeautifulSoup installed. You need to use the same executable or create a virtual environment from *this* executable, install the required modules in the virtual environment and point the shebang to the Python executable within the virtual environment. — Booboo, Dec 21 '20 at 11:31
@Booboo thanks, If you want you can make an answer out of this and I may accept it if it solves the problem — java, Dec 21 '20 at 11:35

Booboo · Accepted Answer · 2020-12-21T11:46:16.500

If it is not the same Python executable, that's a problem, You are therefore running an installation of Python that doesn't have BeautifulSoup installed. You need to use the same executable or create a virtual environment from this executable, install the required modules in the virtual environment and point the shebang to the Python executable within the virtual environment.

For example, to use the /usr/bin/python3 executable with a virtual environment that has your modules:

cd /home/usr/mydir
/usr/bin/python3 -m venv my_venv
cd my_venv/bin
. activate
pip install bs4
deactivate

Then change shebang to point to:

#!/home/usr/mydir/my_venv/bin/python3

Using CGI - python having problem finding modules

1 Answers1