1

Here's what happens when I try to uninstall a package:

(soothsayer_py3.8_env) jespinozlt2-osx:Prototype jespinoz$ pip uninstall soothsayer
Found existing installation: soothsayer 2020.5.1
Uninstalling soothsayer-2020.5.1:
  Would remove:
    /Users/jespinoz/anaconda3/envs/soothsayer_py3.8_env/bin/clairvoyance.py
    /Users/jespinoz/anaconda3/envs/soothsayer_py3.8_env/bin/run_soothsayer.py
    /Users/jespinoz/anaconda3/envs/soothsayer_py3.8_env/lib/python3.8/site-packages/
  Would not remove (might be manually added):
    /Users/jespinoz/anaconda3/envs/soothsayer_py3.8_env/lib/python3.8/site-packages/soothsayer-2020.5.1.dist-info/Icon

Here's the setup.py:

import re, datetime
from setuptools import setup, find_packages

# Version
version = None
with open("./soothsayer/__init__.py", "r") as f:
    for line in f.readlines():
        line = line.strip()
        if line.startswith("__version__"):
            version = line.split("=")[-1].strip().strip('"')
assert version is not None, "Check version in soothsayer/__init__.py"

setup(name='soothsayer',
      version=version,
      description='High-level API for (bio-)informatics',
      url='https://github.com/jolespin/soothsayer',
      author='Josh L. Espinoza',
      author_email='jespinoz@jcvi.org',
      license='BSD-3',
      packages=find_packages(include=("*", "./*")),
      install_requires=[
    "matplotlib >= 2.2.2",
    "scipy >= 1.0",
    "scikit-learn >= 0.20.2",
    "numpy >= 1.13", #, < 1.14.0",
    'pandas >= 1.0',
    'networkx >= 2.0',
    'ete3 >= 3.0',
    'scikit-bio >= 0.5.1',
    "biopython >= 1.5",
    "xarray >= 0.10.3",
    "tqdm >=4.19",
    "openpyxl >= 2.5",
    # "astropy >= 3.0",
    "rpy2 >= 2.9.4, < 3.0",
    "matplotlib_venn",
    "palettable >= 3.0.0",
    "adjustText",
    "tzlocal",
    "statsmodels >= 0.10.0", #https://github.com/statsmodels/statsmodels/issues/5899 May need to use this separately: pip install git+https://github.com/statsmodels/statsmodels.git@maintenance/0.10.x
    "teneto",
    "mmh3",
    "soothsayer_utils >= 2020.4.30",

      ],
     include_package_data=True,
     scripts=['bin/clairvoyance.py', "bin/run_soothsayer.py"],


)

Here's my manifest.in:

recursive-include soothsayer/io/data_type/ *.py
recursive-include soothsayer/feature_extraction/algorithms/ *.py
recursive-include soothsayer/r_wrappers/packages/ *.py
recursive-include soothsayer/tests/data/ *.py
recursive-include soothsayer/db/ *.pbz2
recursive-exclude releases/ *.tar.gz
global-exclude *.py[cod] __pycache__

Here's my pip version: 

pip 20.1 from /Users/jespinoz/anaconda3/envs/soothsayer_py3.8_env/lib/python3.8/site-packages/pip (python 3.8)

I can't seem to figure it out? Does anyone know what could be causing this issue?

O.rka
  • 29,847
  • 68
  • 194
  • 309
  • 2
    This seems like a question that would be better suited for [pip's issue tracker](https://github.com/pypa/pip/issues). – Dustin Ingram May 04 '20 at 19:57
  • What is the output of `path/to/pythonX.Y -m pip show --files soothsayer`? ... OK, I installed this package locally, and had a look at it, and there are some unusual things going on with your packaging. My first thought is to inspect this line: `packages=find_packages(include=("*", "./*")),`. I will look into this further. – sinoroc May 05 '20 at 07:44

1 Answers1

0

I assume the PyPI page and the source code repository for the project causing the issue are the following:

There is a file with a name containing unusual characters at the root of the source code repository. It sometimes appears as something like 'Icon'$'\r'. That file seems to be also present in the source distribution for soothsayer 2020.5.4 in the soothsayer.egg-info directory. So when pip installs that distribution, it somehow makes strange records of files being installed:

$ pip show --files soothsayer
Name: soothsayer
Version: 2020.5.4
Summary: High-level package for (bio-)informatics
Home-page: https://github.com/jolespin/soothsayer
Author: Josh L. Espinoza
Author-email: jespinoz@jcvi.org
License: BSD-3
Location: /tmp/tmp.YDoTgEDb9A/.venv/lib/python3.6/site-packages
Requires: networkx, teneto, palettable, matplotlib-venn, rpy2, xarray, ete3, biopython, tqdm, pandas, scikit-learn, scipy, numpy, scikit-bio, openpyxl, mmh3, matplotlib, statsmodels, adjustText, soothsayer-utils, tzlocal
Required-by: 
Files:
  "
  "soothsayer-2020.5.4.dist-info/Icon
  .
  ../../../bin/__pycache__/clairvoyance.cpython-36.pyc
  ../../../bin/__pycache__/run_soothsayer.cpython-36.pyc
  ../../../bin/clairvoyance.py
  ../../../bin/run_soothsayer.py
  soothsayer-2020.5.4.dist-info/INSTALLER
  soothsayer-2020.5.4.dist-info/Icon
  soothsayer-2020.5.4.dist-info/METADATA
  soothsayer-2020.5.4.dist-info/RECORD
  soothsayer-2020.5.4.dist-info/WHEEL
  soothsayer-2020.5.4.dist-info/top_level.txt
  soothsayer/__init__.py

In the above, note the 3 first recorded Files. These records are obviously wrong. In particular the 3rd line would probably lead pip to intent to delete the site-packages directory.

I would recommend the owner of that soothsayer project should:

  • figure out how such files were created and disable the software that created these files
  • clean up the source code repository in all branches to prevent this from happening in the future releases
  • publish maintenance or bug-fix releases for all faulty releases
  • remove all faulty distributions published on PyPI and other similar indexes

Update

After further inspection, it seems like there is a similar unusually named Icon file in each directory of the source code repository, those would most likely need to disappear as well (in addition to many other unusual things lying around in that repository).

sinoroc
  • 18,409
  • 2
  • 39
  • 70
  • Thanks for identifying this. I'm creating these on OSX and I believe these files are added to each directory by default. Is there a way to exclude the .pyc files and Icon files? I create my distribution by `python setup.py sdist`. – O.rka May 05 '20 at 19:05
  • To be honest, I don't know how such files ended in the _sdist_. I would need to try and reproduce it. Also, if these are more or less standard Mac OS files, I am surprised if they had file names that would cause such effects. I am a bit confused, as to how this could happen. You'd need to figure out what the file names actually are, whether or not you need to add them to `.gitignore` and to some exclusion rules in `MANIFEST.in`. – sinoroc May 05 '20 at 20:37
  • Is the MANIFEST.in used by PyPi? It's definitely something I'm doing because this has happened with another package I've created. – O.rka May 06 '20 at 05:10
  • In short, `MANIFEST.in` helps including or excluding files in the _source distribution_ (`sdist`). But as far as I know by default, _setuptools_ wouldn't include such files in the sdist, so I don't see the need for adding an explicit exclusion rule in `MANIFEST.in`. So I am really not sure how this could have happened, have never seen anything like it, but I don't use that OS. It'd probably be interesting to first figure out why these files are in your filesystem and try to deactivate their creation. There is a limit to files one should exclude with `.gitignore` and `MANIFEST.in`. – sinoroc May 06 '20 at 07:00
  • Can you add how I can add this into my .gitignore and MANIFEST.in files? I'll try it and select it as answered. – O.rka May 14 '20 at 21:06
  • I don't think I will do that. After thinking about it, I do not believe such files should be listed in `.gitignore` or `MANIFEST.in`, I believe that would be somewhat counter productive. There seems to be an issue on the operating system used to create the git repository (and the _sdist_), and I believe this is where the issue should be fixed, not in `.gitignore` or `MANIFEST.in`. The best thing you could do, is figure out which software created these files and disable it, then clean up everything (including on _PyPI_ and in _git_ repository). – sinoroc May 15 '20 at 07:13