1

So I have published a conda package (link).

This package contains .c extensions (coming from cython code), which need to be compiled when the package is installed. My problem is that none of the extensions are compiled when running the install command

conda install -c nicolashug scikit-surprise

Compiling the extensions can be done by simply running

python setup.py install

which is exactly what pip does. The package is on PyPI and works fine.

As far as I understand, this setup.py command is only called when I build the conda package using conda build: the meta.yaml file (created with conda skeleton) contains

build:
    script: python setup.py install  --single-version-externally-managed--record=record.txt

But I need this to be done when the package is installed, not built.

Reading the conda docs, it looks like the install process is merely a matter of copying files:

Installing the files of a conda package into an environment can be thought of as changing the directory to an environment, and then downloading and extracting the .zip file and its dependencies

That would mean I would have to build the package for all platforms and architectures, and then upload them to conda... Which is impossible to me.

So, is there a way to build the package when it is installed, just like pip does?

merv
  • 67,214
  • 13
  • 180
  • 245
Niourf
  • 450
  • 1
  • 4
  • 15
  • 1
    The whole idea of a conda package is that *you* do the compiling so that I don't have to on my machine, and all that's distributed is the compiled library etc.. Is there a particular reason you need the C code compiled on the user's machine at install? If you are worried about multiple platforms, you can a) distribute packages only for the platforms you have access to, or b) set up some continuous integration services (Travis CI, Appveyor, CircleCI) to perform the build and upload for you – darthbith Oct 25 '17 at 13:46
  • I need the code to be compiled on the user's machine for compatibility reasons: the extension compiled on my machine (linux-64) won't work on OSX or Windows, and I don't have access to any other platform (nor do I want to compile on each platform...) If the only way is to use CI services then I guess I'll just tell users to install the package with pip. Thanks! – Niourf Oct 25 '17 at 15:51
  • It looks like you already uploaded packages for each platform? As I said, the idea of conda is (especially for Windows users) that they just get a compiled library. On Windows, setting up compilers so they work properly is a big big PITA, which is one of the biggest reasons for conda (and wheels installed by pip). I'm not trying to tell you how to distribute your code :-) but just pointing out that users on some platforms might have a hard time pip-installing a package with compiled extensions. – darthbith Oct 25 '17 at 16:05
  • yes I uploaded the package for all platforms but that was before I realized that none of them were working, except for the linux-64 ones ^^. "not trying to tell you how to distribute your code" -> I know! Thanks a lot for your help – Niourf Oct 25 '17 at 16:43
  • 1
    "I need the code to be compiled on the user's machine for compatibility reasons". No, you don't. You need the code to be compiled on a compatible Windows platform in order to make a windows conda package. That's done by CI in order to generate the conda package. If you need cython to re-generate the C source code for windows compatibility then you need to do that *before* building the conda package, not when the package is installed. This goes for pip binary wheels also. – danny Oct 26 '17 at 10:39
  • I perfectly understand that. It's exactly what darthbith said... As for the creation of the .c code by Cython, it only need to be done once, whatever the platform: that's what Cython is made for. – Niourf Oct 26 '17 at 14:31

1 Answers1

3

As far as I know, there is no way to have the compilation happen on the user's machine when installing a conda package. Indeed, the whole idea of a conda package is that you do the compiling so that I don't have to on my machine, and all that's distributed is the compiled library. On Windows in particular, setting up compilers so they work properly (with Python) is a big big PITA, which is one of the biggest reasons for conda (and also wheels installed by pip).

If you don't have access to a particular OS directly, you can use Continuous Integration (CI) services, such as Appveyor (Windows), Travis CI (Linux/macOS), or CircleCI (Linux/macOS) to build packages and upload them to Anaconda cloud (or to PyPI for that matter). These services integrate directly with GitHub and other code-sharing services, and are generally free for FOSS projects. That way, you can build packages on each commit, on each tag, or some other variation that you desire.

In the end, you may save more time by setting up these services, because you won't have to provide compiler support for users who can't install a source package from PyPI.

darthbith
  • 18,484
  • 9
  • 60
  • 76