Python Packaging: Creating a dependency on a `conda-forge` package in `conda` `meta.yaml` file

Question

I am writing a package for conda-forge and needed to specify a dependency on another conda-forge dependency. Essentially, I need to install a pinned version of the conda-forge gdal package, because it actually compiles the version of libtiff that supports BIGTIFF files....

Now if I was installing gdal into a conda environment, I would write something like.

conda install -c conda-forge gdal=2.4.4

I would to get this version of gdal=2.4.4 from conda-forge installed when installing the package. Now in the meta.yaml file, I can specify package dependencies like so, but I did not see how to specify a URL to a tar file, or whatever would work.

{% set version = "0.0.1" %}

package:
  name: mypackage
  version: {{ version }}

source:
  url: https://github.com/myrepo/{{ version }}.tar.gz
  sha256: ****6a63

build:
  number: 1
  skip: true  # [win and py27]
  entry_points:
    - mycli = mypackage.main:main

requirements:
  build:
    - python
    - 
  host:
    - python
    - pip
    - numpy
    - gdal  # <----- want to specify from conda-forge
  run:
    - python
    - gdal  # <----- want to specify from conda-forge

Any suggestions about how to do this would be appreciated.

Stuart Berg · Accepted Answer · 2020-08-02T18:29:52.787

I don't think it's possible to specify the channel in meta.yaml. The following issue is still unresolved in the conda-build issue tracker: https://github.com/conda/conda-build/issues/532

As a workaround, if you know the exact version of gdal that you need, you can specify the exact version and "build string" in the recipe.

The only annoying thing is that you'll have to list gdal once for every combination of platform and python version your recipe needs to support.

requirements:
  build:
    - python
    - 
  host:
    - python
    - pip
    - numpy
    - gdal  2.4.4 py36h02fde04_1  # [osx and py==36]
    - gdal  2.4.4 py37h622575a_1  # [osx and py==37]
    - gdal  2.4.4 py38h57202bd_1  # [osx and py==38]
    - gdal  2.4.4 py36hbb8311d_1  # [linux and py==36]
    - gdal  2.4.4 py37hf8c3989_1  # [linux and py==37]
    - gdal  2.4.4 py38hfe926b7_1  # [linux and py==38]

  run:
    - python
    - gdal  2.4.4 py36h02fde04_1  # [osx and py==36]
    - gdal  2.4.4 py37h622575a_1  # [osx and py==37]
    - gdal  2.4.4 py38h57202bd_1  # [osx and py==38]
    - gdal  2.4.4 py36hbb8311d_1  # [linux and py==36]
    - gdal  2.4.4 py37hf8c3989_1  # [linux and py==37]
    - gdal  2.4.4 py38hfe926b7_1  # [linux and py==38]

(I copied those from the gdal package listing on the conda-forge channel.)

BTW, since you mentioned that the real important difference for you is libtiff, then should you be pinning libtiff instead of gdal? Or maybe both?

Edit:

It would be nice to avoid repeating the whole list of build strings in the host and run sections.

As you suggested in the comments, one option is to define the build string in conda_build_config.yaml:

# conda_build_config.yaml
gdal_build:
  - py36h02fde04_1  # [osx and py==36]
  - py37h622575a_1  # [osx and py==37]
  - py38h57202bd_1  # [osx and py==38]
  - py36hbb8311d_1  # [linux and py==36]
  - py37hf8c3989_1  # [linux and py==37]
  - py38hfe926b7_1  # [linux and py==38]

# meta.yaml
requirements:
  build:
    - python
    - 
  host:
    - python
    - pip
    - numpy
    - gdal  2.4.4 {{ gdal_build }}

  run:
    - python
    - gdal  2.4.4 {{ gdal_build }}

Another option is to define a lookup table in a jinja variable, directly in meta.yaml. This is slightly uglier, perhaps, but at least all of logic is contained in a single file. I'm not sure which to prefer.

{% set platform = 'linux' if linux else 'osx' if osx else 'win' %}

{%
  set gdal_builds = {
    'osx': {
      36: 'py36h02fde04_1',
      37: 'py37h622575a_1',
      38: 'py38h57202bd_1',
    },
    'linux': {
      36: 'py36hbb8311d_1',
      37: 'py37hf8c3989_1',
      38: 'py38hfe926b7_1',
    }
  }
%}

requirements:
  build:
    - python
    - 
  host:
    - python
    - pip
    - numpy
    - gdal  2.4.4 {{ gdal_builds[platform][py] }}

  run:
    - python
    - gdal  2.4.4 {{ gdal_builds[platform][py] }}

thanks so much for the help. Yeah, GDAL is so sensitive, that it is hard to find a build that works. I could probably look at the GDAL build that I use on `conda-forge` and try to pin those versions. The specific version of `libtiff` on the `conda-forge` version of `gdal` supports the BIGTIFF format--for raster images that can exceed 4GB. This is a good start through, and that is usually the hardest part. So thanks again. — krishnab, Jul 31 '20 at 04:23
Okay so if I use the actual build string, does that mean `conda` will look through the `defaults` channel, not find the precise build string, and then it will look in `conda-forge` for the exact build string, find it, and install it? Is that the way that this works. Just wanted to make sure I understood. — krishnab, Aug 01 '20 at 20:02
@user162094 Also would it make sense to save these variants in the `conda_build_config.yaml` and then feed then in through a variable? Or is it better to just keep them all there in the `meta.yaml` file. — krishnab, Aug 01 '20 at 20:04
You are correct about how this example works. It will search for the names in both conda forge and defaults, and the order in which it searches depends on the channel priority in your .condarc file. — Stuart Berg, Aug 02 '20 at 16:48
@krishnab I like your suggestion to use a variable to store the build string. I can think of two ways to do that. Please see my edits above. — Stuart Berg, Aug 02 '20 at 18:30
yes this is really nice. I was thinking the first choice, but I did not know you could do the second choice. So it is really nice to see the difference. This is great though. I need to make sure I record this code sample for my own records. — krishnab, Aug 02 '20 at 20:35

Python Packaging: Creating a dependency on a `conda-forge` package in `conda` `meta.yaml` file

1 Answers1

Linked

Related