4

TL;DR: Running python setup.py develop --uninstall from a pypy environment created using tox results in an exception: error: [Error 32] The process cannot access the file because it is being used by another process: c:\users\shach\code\pydocstyle\.tox\pypy\site-packages\funniest.egg-link.

Hello folks,

I have a set of integration tests for my python package that do the following:

  1. Call python setup.py develop (using subprocess.check_call)
  2. Run all the tests for the package
  3. Call python setup.py develop --uninstall (again, using subprocess.check_call)

The tests are being run by tox. Everything works fine on python versions 27, 33, 34, 35 and 36, but the code fails on pypy.

I'm not going to include the setup.py and project file here, you can assume they're fine. I've reproduced this using the minimal package shown here and it works (fails?) all the same.

To reproduce, I created a python script that runs the following:

import shlex
import subprocess

subprocess.check_call(shlex.split('python setup.py develop'))
print('----------')
subprocess.check_call(shlex.split('python setup.py develop --uninstall'))

Running the file using a regular pypy installed on the system works just fine:

C:\Users\shach\code\bla\funniest>pypy test.py
running develop
running egg_info
writing funniest.egg-info\PKG-INFO
writing dependency_links to funniest.egg-info\dependency_links.txt
writing top-level names to funniest.egg-info\top_level.txt
reading manifest file 'funniest.egg-info\SOURCES.txt'
writing manifest file 'funniest.egg-info\SOURCES.txt'
running build_ext
Creating c:\python\pypy\site-packages\funniest.egg-link (link to .)
Adding funniest 0.1 to easy-install.pth file

Installed c:\users\shach\code\bla\funniest
Processing dependencies for funniest==0.1
Finished processing dependencies for funniest==0.1
----------
running develop
Removing c:\python\pypy\site-packages\funniest.egg-link (link to .)
Removing funniest 0.1 from easy-install.pth file

C:\Users\shach\code\bla\funniest>

But when I run it from a tox environment for pypy:

C:\Users\shach\code\bla\funniest>tox
GLOB sdist-make: C:\Users\shach\code\bla\funniest\setup.py
pypy inst-nodeps: C:\Users\shach\code\bla\funniest\.tox\dist\funniest-0.1.zip
pypy installed: cffi==1.10.1,funniest==0.1,greenlet==0.4.12,readline==6.2.4.1
pypy runtests: PYTHONHASHSEED='122'
pypy runtests: commands[0] | python test.py
running develop
running egg_info
writing funniest.egg-info\PKG-INFO
writing dependency_links to funniest.egg-info\dependency_links.txt
writing top-level names to funniest.egg-info\top_level.txt
reading manifest file 'funniest.egg-info\SOURCES.txt'
writing manifest file 'funniest.egg-info\SOURCES.txt'
running build_ext
Creating c:\users\shach\code\bla\funniest\.tox\pypy\site-packages\funniest.egg-link (link to .)
funniest 0.1 is already the active version in easy-install.pth

Installed c:\users\shach\code\bla\funniest
Processing dependencies for funniest==0.1
Finished processing dependencies for funniest==0.1
----------
running develop
Removing c:\users\shach\code\bla\funniest\.tox\pypy\site-packages\funniest.egg-link (link to .)
error: [Error 32] The process cannot access the file because it is being used by another process: c:\users\shach\code\bla\funniest\.tox\pypy\site-packages\funniest.egg-link
Traceback (most recent call last):
  File "test.py", line 6, in <module>
    subprocess.check_call(shlex.split('python setup.py develop --uninstall'))
  File "C:\Python\pypy\lib-python\2.7\subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['python', 'setup.py', 'develop', '--uninstall']' returned non-zero exit status 1
ERROR: InvocationError: 'C:\\Users\\shach\\code\\bla\\funniest\\.tox\\pypy\\bin\\python.EXE test.py'
___________________________________ summary ___________________________________
ERROR:   pypy: commands failed

C:\Users\shach\code\bla\funniest>

Here's the tox.ini I'm using to reproduce:

[tox]
envlist = pypy
[testenv]
commands=python test.py

I made sure that I have read/write permissions to the directory, and am kinda losing my mind here.

It works just fine on linux. Probably because you can remove a file that's being used (inodes, etc.) :^)

UPDATE 1:

I thought maybe the problem lies in that both develop and develop --uninstall are being run from the same python file, and that some resource is not being cleaned properly (perhaps an open file that locks the egg-link), so i ran it manually:

C:\Users\shach\code\bla\funniest>.tox\pypy\bin\activate.bat

(pypy) C:\Users\shach\code\bla\funniest>pypy setup.py develop
running develop
running egg_info
writing funniest.egg-info\PKG-INFO
writing dependency_links to funniest.egg-info\dependency_links.txt
writing top-level names to funniest.egg-info\top_level.txt
reading manifest file 'funniest.egg-info\SOURCES.txt'
writing manifest file 'funniest.egg-info\SOURCES.txt'
running build_ext
Creating c:\users\shach\code\bla\funniest\.tox\pypy\site-packages\funniest.egg-link (link to .)
funniest 0.1 is already the active version in easy-install.pth

Installed c:\users\shach\code\bla\funniest
Processing dependencies for funniest==0.1
Finished processing dependencies for funniest==0.1

(pypy) C:\Users\shach\code\bla\funniest>pypy setup.py develop --uninstall
running develop
Removing c:\users\shach\code\bla\funniest\.tox\pypy\site-packages\funniest.egg-link (link to .)
error: [Error 32] The process cannot access the file because it is being used by another process: c:\users\shach\code\bla\funniest\.tox\pypy\site-packages\funniest.egg-link

(pypy) C:\Users\shach\code\bla\funniest>

Still fails :(

Please advise!

Shachar.

UPDATE 2:

I've tried to check if there's a process holding that file using Sysinternal's Process Explorer, and nada. I manually ran pypy setup.py develop, then checked in Process Explorer and made sure the file is not opened in any process, and then ran pypy setup.py develop --uninstall and the same error was raised.

UPDATE 3:

Still happens after closing CMD and opening a new one.

UPDATE 4:

REBOOTING THE MACHINE DOES NOT SOLVE THIS! Wha-----?

Shachar Ohana
  • 162
  • 8
  • Can you explain why you run the install command in a subprocess and also why you run the uninstall command at all? I don't know why the particular problem occurs in pypy, but why not use [usedevelop](http://tox.readthedocs.io/en/latest/config.html#confval-usedevelop=BOOL) and be done with it (and if you want to clean up run `tox ---recreate`). – Oliver Bestwalter Sep 25 '17 at 13:40
  • The project has many tests, some are unit tests and some are integration tests. I want the integration tests to actually run my program - the setup.py has a console script entry point that needs to be tested. In my integration tests I have a py.test fixture that runs `develop` before the tests, and `develop -u` after them. I don't want to leave the package installed, and running tox -r every time takes a lot of time. – Shachar Ohana Sep 25 '17 at 13:47
  • oh - o.k. thanks for the explanation. It just sounded strange out of context. For some sophisticated setups it might make sense, though I remain sceptical if that approach is the best way to go about it, as you are actively working against what tox provides. Maybe extra environments doing those tests, would be a better approach (running them in parallel then with e.g. [detox](https://github.com/tox-dev/detox)). – Oliver Bestwalter Sep 25 '17 at 13:50
  • I agree there are other solutions that will give the same result, but I'm trying to stick to that one in the meantime because it gives me the desired result in the CI - I have a build for each python version for both linux and windows and I can tell in O(1) what works and what doesn't. `detox` won't be much of a help since it will only speed up multiple envs tested on the same `tox` command, which is not the case here. The underlying cause for my bug really bugs me (ha ha). How can a file be locked without any process locking it? o.O – Shachar Ohana Sep 25 '17 at 14:01
  • Yeah, o.k. - I understand that. The problem you describe leaves me comepletely clueless - I have not much experience with pypy whatsoever - so won't be able to help you with that, sorry. I wanted to make sure that you see potential alternatives that completely circumvent the original problem :) – Oliver Bestwalter Sep 25 '17 at 14:05

2 Answers2

1

From the description it would seem that somewhere in the running of pypy setup.py develop --uninstall, a file or directory is being held open, which prevents its deletion on windows. You can try to run pypy with "-X track-resources" argument (pypy -X track-resources setup.py develop --uninstall) which should notify you if an object holding resources is deallocated before the resource is closed. Sometimes scanning for code like s - open(path).read() may be enough to find the problem, that code will leak the file descriptor until the garbage collector runs, notices the ghost object, and deletes it. The code should use a context manager; with open(path) as fid: s = fid.read()

mattip
  • 2,360
  • 1
  • 13
  • 13
  • Sadly nothing appears to be leaky... :( I even dug through `setuptool`'s source code a bit and didn't find any leak in the part that handles the egg links. Ugh. – Shachar Ohana Sep 27 '17 at 20:13
0

TL;DR: Don't use setup.py directly, use pip install -e ..

It seems that the problem is in setuptools on windows. What happens is that the egg-link file is opened&closed just a couple of lines before it's being removed - and windows can't handle that.

Somehow pip does handle this case, I'll update when I have more info.

I've opened an issue in setuptools.

Shachar Ohana
  • 162
  • 8