How do you make Python console script entry points work when installed package uses a conda virtual environment?

Question

Problem - Shifting from non-virtual to a conda virtual environment causes console script entry points to be unrecognized.

Background - I recently tried to get religion about using virtual environments for my Python projects. I decided to do this after update to macOS Catalina caused all of my PyCharm projects to show invalid interpreter errors. I thought "What could go wrong throwing one big mess on top of another?" Two days later I could finally run a script again - the worst brick wall I've ever hit. I was not able to find a solution anywhere, so I'm writing my first SO question and my solution to follow, thinking I might finally have something worthy of contributing back to this site I've used so much for so long.

My setup

• OS: macOS Catalina
• Shell: bash (yes, I changed it back after Catalina update and suppressed the nagging 'zsh is now default' message)
• IDE: PyCharm 19.1 Pro
• Anaconda: 4.4.7
• Python: 3.7

Context - I develop several interacting data science packages and locally install these in editable mode as a general practice via:

My_Machine:my_package my_user_name$ pip install -e .

I create python packages using a setup.py file with setuptools, building using PyCharm. Within the setup.py file, I define console script entry points like this:

setup.py:

# -*- coding: utf-8 -*-

from setuptools import setup, find_packages

setup(...
      name='my_project',
      entry_points={'console_scripts':['my_entry_name=my_package.scripts.my_python_script:main'
                                      ]},
     ...
)

Before shifting to a conda virtual environment, I was running script perfectly fine for years via a batch file like this:

my_batch_file.command:

#!/bin/bash
cd "$(dirname "$0")"  # set the working directory as the command file locations

my_entry_name <script arguments>

However, after shifting to a conda virtual environment, running the command file produces a my_entry_name: command not found error.

Things tried so far

• Verified and tried to set which python is used via which python terminal command. I can see that the default is /Users/my_user_name/anaconda3/bin/python and if I do this from the command prompt within my project, I see /Users/my_user_name/anaconda3/envs/my_env/bin/python, reflecting the environment version as expected.
• Checked in the actual entry point file in /Users/my_user_name/anaconda3/envs/my_env/bin/my_entry_name to see how the shebang line indicates the python version, which was #!/Users/my_user_name/anaconda3/envs/my_env/bin/python
• Tried adding this shebang to the top of my .command file
• Reinstalled my package many times, thinking the entry point might not be registered right somehow.
• Messed around with bash vs. zsh a lot, thinking the transition to zsh by Catalina update, and back to bash might have caused problems.
• Tried to get back to functioning by going back from virtual environment, but couldn't get PyCharm non-virtual interpreter settings to work again.
• Looked at $PATH contents for issues.
• Read lots of tutorials about virtual environments (all I found stop at very basics)
• pursued threads about bugs in setuptools related to virtual environments
• Many combinations of these efforts.

None of this worked - same my_entry_name: command not found error. Extremely frustrating two days.

Answer 1

After two days of struggling, I found a solution, which I will post below, however I would very much like to hear comments, alternatives, and straight up 'you're an idiot' schooling.

It seems two elements are necessary:

• Indicating the version of python to use as the one associated with the virtual environment (as I had tried unsuccessfully).
• Activating the virtual environment.

I changed my .command file to this, adding several unnecessary verification checks of the python version before and after specifying it and associated text output:

my_batch.command:

#!/usr/bin/env bash

echo "VERIFY: Python version BEFORE activating virtual environment:"
which python

echo "Activating conda virtual environment..."
source activate my_env_name

echo "Setting python version to use in this environment..."
#!/Users/my_username/anaconda3/envs/my_env_name/bin/python

echo "VERIFY: Python version AFTER activating virtual environment:"
which python

cd "$(dirname "$0")"  # set the working directory as the command file locations

echo "RUN THE SCRIPT:"
my_entry_name <script arguments>

The key line that I was missing was the source activate my_env_name. I verified that removing this causes failure, and this must be included each time, not just once, thus included in my .command file.

It was also not clear to me that it was ok to have multiple shebang lines, but this worked fine.

I'm glad to be functioning again, but I have to admit I'm frustrated that the architecture doesn't just make my entry point work, period, without this clunkiness. The reason to have the entry point is to allow a consumer of the script to call it easily and not have to care about details like where the script is and how it's installed. The virtual environment usage seems to eliminate these entry point conveniences. I'm all for the benefits of using virtual environments, but is there some way to have my cake and eat it too? Ideally calling the entry point would activate the virtual environment and know which version of python to use. Is there some better way to do this that does that?

Answer 2

You don't have to activate Python virtual environments, ever, not even once. Say you have a virtual environment at /venv, then you can call /venv/bin/python or /venv/bin/my_entry_name from anywhere anytime without activating the virtual environment and it should work perfectly fine. If it doesn't work then something is wrong with your setup that has to be fixed.

When you use setuptools console_scripts entry points the full path to the Python interpreter within the virtual environment is hard-coded in the shebang, so there should be no way this doesn't work.

---

Update

My bad, I had not registered that you are working with conda virtual environments. I don't know how these work, maybe my point still stands, but probably not.

Answer 3

Revised answer after further input and experimenting. I've found two options where both require knowing where the actual file is that defines the entry point. For a conda virtual environment, it will be here:

/anaconda3/envs/my_env_name/bin/entry_point_name

There is no need to specify the python version or to activate the environment if this file is called. I've found two ways to do this:

Option 1 - Credit to @sinoroc who first pointed out the virtual environment does not need to be activated if the entry point is called correctly. For a conda virtual environment, that would be:

my_batch_file.command

#!/bin/bash
cd "$(dirname "$0")"  # set the working directory as the command file locations

~/anaconda3/envs/my_env_name/bin/entry_point_name <my script args>

For other types of virtual environments, you would just adjust the path details to get to the entry point file.

Option 2 - If you place a copy of the entry point file in the main Anaconda bin:

/anaconda3/bin/my_entry_name

you don't need to specify the path, letting you call the entry point like they should work, imho - ie shields the script user from language and install details:

my_batch_file.command

#!/bin/bash
cd "$(dirname "$0")"  # set the working directory as the command file locations

entry_point_name <my script args>

As this manual copy step is not elegant, I have posted a question about how to do this better: How do you make an entry point to a script in a virtual environment available system-wide?