I tried to build and install a simple python package using conda-build and conda install and read many documentions on the internet such as the following:
https://conda.io/projects/conda-build/en/latest/user-guide/tutorials/build-pkgs.html https://docs.conda.io/projects/conda-build/en/latest/user-guide/environment-variables.html https://docs.conda.io/projects/conda-build/en/latest/resources/build-scripts.html https://docs.conda.io/projects/conda-build/en/latest/resources/commands/conda-build.html https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html
My expection and goal is that after building the tarball using conda-build and then running:
cd opt/anaconda3/conda-bld/noarch
conda activate test_env
conda install --use-local my_test_pkg-0.5.0-py_0.tar.bz2
that I find my package in opt/anaconda3/envs/python3.8/lib/site-packages/my_test_pkg. However, after the installation finishes the tarball my_test_pkg-0.5.0-py_0.tar.bz2 is only moved to opt/anaconda3/pkgs and extracted there. But there are no files in any site-packages folder of any base or virtual environment. I took look at how other tarballs look like, e.g. pandas. There I found that next to a info directory there is also a lib directory. I suppose that the directory in the tarball needs to match the directory where its content is being distruted to. I took a look at the meta.yaml in pandas but I could not find any info how this "lib" directory in the tarball is created. Then I tried to manually package and created the same structure as it is in pandas and ran "conda install my_test_pkg-0.5.0-py_0.tar.bz2" again but nothing really happened. So what does conda install actually do?
This is how my meta.yaml looks like
{% set version = "0.5.0" %}
package:
name: my_test_pkg
version: {{ version }}
source:
- path: my_test_pkg
build:
noarch: python
number: 0
requirements:
host:
- python >=3.8
- conda >=4.10
- conda-build
- anaconda-client
run:
- python >=3.8
- matplotlib >=3.3.2
- numpy >=1.18
- pandas >=1.1.2
about:
license: GNU General Public License (GPL)
summary: 'A package for testing the conda build mechanisms'
And this is my build.sh
#!/bin/sh
# Variables
# Actually $CONDA_PREFIX was intended to be used here but it seems that it is overwritten with $BUILD_PREFIX.
dir_in_conda=$SYS_PREFIX/lib/python3.8/site-packages/$PKG_NAME
# Install
echo "Start installation"
# mkdir $dir_in_conda
# cp -RT $RECIPE_DIR/$PKG_NAME $dir_in_conda
As you see I tried also to do the installation in the build.sh script but then I do not know what the "conda install" command is actually for. The output of the conda install command was:
conda install --use-local my_test_pkg-0.5.0-py_0.tar.bz2
Downloading and Extracting Packages
my_test_pkg-0.5.0 | ################################################# | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
So what do I need to do and where in order to have my_test_package properly distributed to the conda virtual environment?
UPDATE: I saw that my info/files and info.paths.json files are empty. But why? I must be doing something wrong during the build process.
Full stack trace of build process
No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.16
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.16
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Attempting to finalize metadata for my_test_pkg
INFO:conda_build.metadata:Attempting to finalize metadata for my_test_pkg
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
BUILD START: ['my_test_pkg-0.5.0-py_0.tar.bz2']
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
## Package Plan ##
environment location: /home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place
The following NEW packages will be INSTALLED:
_libgcc_mutex: 0.1-conda_forge conda-forge
_openmp_mutex: 4.5-1_gnu conda-forge
anaconda-client: 1.7.2-py_0 conda-forge
attrs: 20.3.0-pyhd3deb0d_0 conda-forge
beautifulsoup4: 4.9.3-pyhb0f4dca_0 conda-forge
brotlipy: 0.7.0-py39h3811e60_1001 conda-forge
bzip2: 1.0.8-h7f98852_4 conda-forge
ca-certificates: 2020.12.5-ha878542_0 conda-forge
certifi: 2020.12.5-py39hf3d152e_1 conda-forge
cffi: 1.14.5-py39he32792d_0 conda-forge
chardet: 4.0.0-py39hf3d152e_1 conda-forge
clyent: 1.2.2-py_1 conda-forge
conda: 4.10.0-py39hf3d152e_1 conda-forge
conda-build: 3.21.4-py39hf3d152e_0 conda-forge
conda-package-handling: 1.7.2-py39h38d8fee_0 conda-forge
cryptography: 3.4.7-py39hbca0aa6_0 conda-forge
filelock: 3.0.12-pyh9f0ad1d_0 conda-forge
glob2: 0.7-py_0 conda-forge
icu: 68.1-h58526e2_0 conda-forge
idna: 2.10-pyh9f0ad1d_0 conda-forge
importlib-metadata: 3.10.0-py39hf3d152e_0 conda-forge
ipython_genutils: 0.2.0-py_1 conda-forge
jinja2: 2.11.3-pyh44b312d_0 conda-forge
jsonschema: 3.2.0-pyhd8ed1ab_3 conda-forge
jupyter_core: 4.7.1-py39hf3d152e_0 conda-forge
ld_impl_linux-64: 2.35.1-hea4e1c9_2 conda-forge
libarchive: 3.5.1-h3f442fb_1 conda-forge
libffi: 3.3-h58526e2_2 conda-forge
libgcc-ng: 9.3.0-h2828fa1_18 conda-forge
libgomp: 9.3.0-h2828fa1_18 conda-forge
libiconv: 1.16-h516909a_0 conda-forge
liblief: 0.10.1-he1b5a44_2 conda-forge
libstdcxx-ng: 9.3.0-h6de172a_18 conda-forge
libxml2: 2.9.10-h72842e0_3 conda-forge
lz4-c: 1.9.3-h9c3ff4c_0 conda-forge
lzo: 2.10-h516909a_1000 conda-forge
markupsafe: 1.1.1-py39h3811e60_3 conda-forge
nbformat: 5.1.3-pyhd8ed1ab_0 conda-forge
ncurses: 6.2-h58526e2_4 conda-forge
openssl: 1.1.1k-h7f98852_0 conda-forge
patchelf: 0.11-he1b5a44_0 conda-forge
pip: 21.0.1-pyhd8ed1ab_0 conda-forge
pkginfo: 1.7.0-pyhd8ed1ab_0 conda-forge
psutil: 5.8.0-py39h3811e60_1 conda-forge
py-lief: 0.10.1-py39hfbd4741_2 conda-forge
pycosat: 0.6.3-py39h3811e60_1006 conda-forge
pycparser: 2.20-pyh9f0ad1d_2 conda-forge
pyopenssl: 20.0.1-pyhd8ed1ab_0 conda-forge
pyrsistent: 0.17.3-py39h3811e60_2 conda-forge
pysocks: 1.7.1-py39hf3d152e_3 conda-forge
python: 3.9.2-hffdb5ce_0_cpython conda-forge
python-dateutil: 2.8.1-py_0 conda-forge
python-libarchive-c: 2.9-py39hf3d152e_2 conda-forge
python_abi: 3.9-1_cp39 conda-forge
pytz: 2021.1-pyhd8ed1ab_0 conda-forge
pyyaml: 5.4.1-py39h3811e60_0 conda-forge
readline: 8.0-he28a2e2_2 conda-forge
requests: 2.25.1-pyhd3deb0d_0 conda-forge
ripgrep: 12.1.1-h516909a_1 conda-forge
ruamel_yaml: 0.15.80-py39h3811e60_1004 conda-forge
setuptools: 49.6.0-py39hf3d152e_3 conda-forge
six: 1.15.0-pyh9f0ad1d_0 conda-forge
soupsieve: 2.0.1-py_1 conda-forge
sqlite: 3.35.4-h74cdb3f_0 conda-forge
tk: 8.6.10-h21135ba_1 conda-forge
tqdm: 4.60.0-pyhd8ed1ab_0 conda-forge
traitlets: 5.0.5-py_0 conda-forge
tzdata: 2021a-he74cb21_0 conda-forge
urllib3: 1.26.4-pyhd8ed1ab_0 conda-forge
wheel: 0.36.2-pyhd3deb0d_0 conda-forge
xz: 5.2.5-h516909a_1 conda-forge
yaml: 0.2.5-h516909a_0 conda-forge
zipp: 3.4.1-pyhd8ed1ab_0 conda-forge
zlib: 1.2.11-h516909a_1010 conda-forge
zstd: 1.4.9-ha95c52a_0 conda-forge
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Copying /home/filip/Dokumente/Programming/build_python_package/my_test_pkg/my_test_pkg to /home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work/
source tree in: /home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work
export PREFIX=/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_place
export BUILD_PREFIX=/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/_build_env
export SRC_DIR=/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work
Start installation
Resource usage statistics from building my_test_pkg:
Process count: 1
CPU time: Sys=0:00:00.0, User=-
Memory: 3.3M
Disk usage: 36B
Time elapsed: 0:00:02.1
Packaging my_test_pkg
INFO:conda_build.build:Packaging my_test_pkg
INFO conda_build.build:build(2274): Packaging my_test_pkg
Packaging my_test_pkg-0.5.0-py_0
INFO:conda_build.build:Packaging my_test_pkg-0.5.0-py_0
INFO conda_build.build:bundle_conda(1514): Packaging my_test_pkg-0.5.0-py_0
No files or script found for output my_test_pkg
WARNING:conda_build.build:No files or script found for output my_test_pkg
WARNING conda_build.build:bundle_conda(1613): No files or script found for output my_test_pkg
number of files: 0
Fixing permissions
INFO :: Time taken to mark (prefix)
0 replacements in 0 files was 0.00 seconds
TEST START: /home/filip/Dokumente/Programming/build_python_package/conda-build/noarch/my_test_pkg-0.5.0-py_0.tar.bz2
Adding in variants from /tmp/tmpw7306v3x/info/recipe/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /tmp/tmpw7306v3x/info/recipe/conda_build_config.yaml
INFO conda_build.variants:_combine_spec_dictionaries(234): Adding in variants from /tmp/tmpw7306v3x/info/recipe/conda_build_config.yaml
Nothing to test for: /home/filip/Dokumente/Programming/build_python_package/conda-build/noarch/my_test_pkg-0.5.0-py_0.tar.bz2
Renaming work directory '/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work' to '/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work_moved_my_test_pkg-0.5.0-py_0_linux-64_main_build_loop'
INFO:conda_build.utils:Renaming work directory '/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work' to '/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work_moved_my_test_pkg-0.5.0-py_0_linux-64_main_build_loop'
INFO conda_build.utils:shutil_move_more_retrying(2077): Renaming work directory '/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work' to '/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work_moved_my_test_pkg-0.5.0-py_0_linux-64_main_build_loop'
shutil.move(work)=/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work, dest=/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work_moved_my_test_pkg-0.5.0-py_0_linux-64_main_build_loop)
INFO:conda_build.utils:shutil.move(work)=/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work, dest=/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work_moved_my_test_pkg-0.5.0-py_0_linux-64_main_build_loop)
INFO conda_build.utils:shutil_move_more_retrying(2084): shutil.move(work)=/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work, dest=/home/filip/Dokumente/Programming/build_python_package/conda-build/my_test_pkg_1617828659204/work_moved_my_test_pkg-0.5.0-py_0_linux-64_main_build_loop)
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:
# To have conda build upload to anaconda.org automatically, use
# conda config --set anaconda_upload yes
anaconda upload \
/home/filip/Dokumente/Programming/build_python_package/conda-build/noarch/my_test_pkg-0.5.0-py_0.tar.bz2
anaconda_upload is not set. Not uploading wheels: []
INFO :: The inputs making up the hashes for the built packages are as follows:
{
"my_test_pkg-0.5.0-py_0": {
"recipe": {}
}
}
####################################################################################
Resource usage summary:
Total time: 0:02:26.9
CPU usage: sys=0:00:00.0, user=0:00:00.0
Maximum memory usage observed: 3.3M
