2

I'm trying to automate testing of my project using gitlab-ci/cd. Therefore I want to use the 'python setup.py test' command. But it's running about half an hour in the gitlab pipeline.

On my local machine the preparation & tests are running successful and in a few seconds. But on the gitlab runner, the installation of the requirements takes over half an hour. I tried to modify my .gitlab-ci.yml and the setup.py file to fix the issue. For example I added an install_requires line in the setup.py file. According to the place where it stops a long time, it has to have something to do with the pandas requirement.

Here is my current .gitlab-ci.yml:

image: python:3.6

stages:
  - test
  - release

before_script:
  - apk add git openssl-dev build-base libffi-dev libxml2-dev libxslt-dev python-dev
  - pip install -U setuptools

test:
  stage: test
  script:
    - echo "Running Tests"
    - python setup.py test
    - echo "Tests finished successfully"
  tags:
    - xyz
  variables:
    GIT_STRATEGY: clone
#...

Here is my setup.py:

import sys
from setuptools import setup


def setup_package():
    needs_sphinx = {'build_sphinx', 'upload_docs'}.intersection(sys.argv)
    sphinx = ['sphinx'] if needs_sphinx else []
    setup(setup_requires=['six', 'pyscaffold>=2.5a0,<2.6a0'] + sphinx,
          use_pyscaffold=True,
          install_requires=['requests == 2.18', 'pandas == 0.23', #...]
    )


if __name__ == "__main__":
    setup_package()

[... Displays that more code is following, which is propably not relevant.]

The part in gitlab-ci where it takes a long time has a lot of code, so I try to post the important lines:

$ python setup.py test
running test
Searching for requests
Reading https://pypi.org/simple/requests/
Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl#sha256=7bf2a778576d825600030a110f3c0e3e8edc51dfaafe1c146e39a2027784957b
Best match: requests 2.21.0
Processing requests-2.21.0-py2.py3-none-any.whl
Installing requests-2.21.0-py2.py3-none-any.whl to /builds/analytics/data_utils/.eggs
writing requirements to /builds/analytics/data_utils/.eggs/requests-2.21.0-py3.6.egg/EGG-INFO/requires.txt

Installed /builds/analytics/data_utils/.eggs/requests-2.21.0-py3.6.egg
Searching for pandas
Reading https://pypi.org/simple/pandas/
Downloading https://files.pythonhosted.org/packages/b2/4c/b6f966ac91c5670ba4ef0b0b5613b5379e3c7abdfad4e7b89a87d73bae13/pandas-0.24.2.tar.gz#sha256=4f919f409c433577a501e023943e582c57355d50a724c589e78bc1d551a535a2
Best match: pandas 0.24.2
Processing pandas-0.24.2.tar.gz
Writing /tmp/easy_install-au2bizhf/pandas-0.24.2/setup.cfg
Running pandas-0.24.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-au2bizhf/pandas-0.24.2/egg-dist-tmp-76fg254v
/bin/sh: svnversion: not found
non-existing path in 'numpy/distutils': 'site.cfg'
Could not locate executable gfortran
Could not locate executable f95
Could not locate executable ifort
Could not locate executable ifc
Could not locate executable lf95
Could not locate executable pgfortran
Could not locate executable f90
Could not locate executable f77
Could not locate executable fort
Could not locate executable efort
Could not locate executable efc
Could not locate executable g77
Could not locate executable g95
Could not locate executable pathf95
Could not locate executable nagfor
don't know how to compile Fortran code on platform 'posix'
_configtest.c:1:5: warning: conflicting types for built-in function 'exp' [-Wbuiltin-declaration-mismatch]
 int exp (void);
     ^~~

Hundred more warnings

File: build/src.linux-x86_64-3.6/numpy/core/include/numpy/config.h
#define HAVE_ENDIAN_H 1
#define SIZEOF_PY_INTPTR_T 8
#define SIZEOF_OFF_T 8
#define SIZEOF_PY_LONG_LONG 8
#define MATHLIB 
...

In file included from /tmp/easy_install-au2bizhf/pandas-0.24.2/.eggs/numpy-1.16.2-py3.6-linux-x86_64.egg/numpy/core/include/numpy/ndarraytypes.h:1822,
                 from /tmp/easy_install-au2bizhf/pandas-0.24.2/.eggs/numpy-1.16.2-py3.6-linux-x86_64.egg/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /tmp/easy_install-au2bizhf/pandas-0.24.2/.eggs/numpy-1.16.2-py3.6-linux-x86_64.egg/numpy/core/include/numpy/arrayobject.h:4,
                 from pandas/_libs/src/ujson/python/JSONtoObj.c:44:
/tmp/easy_install-au2bizhf/pandas-0.24.2/.eggs/numpy-1.16.2-py3.6-linux-x86_64.egg/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
In file included from pandas/_libs/tslibs/src/datetime/np_datetime.c:24:
/usr/local/include/python3.6m/datetime.h:200:25: warning: 'PyDateTimeAPI' defined but not used [-Wunused-variable]
 static PyDateTime_CAPI *PyDateTimeAPI = NULL;
                         ^~~~~~~~~~~~~
In file included from pandas/_libs/tslibs/src/datetime/np_datetime.h:25,
                 from pandas/_libs/tslibs/src/datetime/np_datetime_strings.c:37:
/usr/local/include/python3.6m/datetime.h:200:25: warning: 'PyDateTimeAPI' defined but not used [-Wunused-variable]
 static PyDateTime_CAPI *PyDateTimeAPI = NULL;
                         ^~~~~~~~~~~~~
UPDATING build/lib.linux-x86_64-3.6/pandas/_version.py
set build/lib.linux-x86_64-3.6/pandas/_version.py to '0.24.2'
creating /builds/analytics/data_utils/.eggs/pandas-0.24.2-py3.6-linux-x86_64.egg
Extracting pandas-0.24.2-py3.6-linux-x86_64.egg to /builds/analytics/data_utils/.eggs

Installed /builds/analytics/data_utils/.eggs/pandas-0.24.2-py3.6-linux-x86_64.egg

At the end the job passes, but it takes over half an hour. I would appreciate any help on how to speed up the processing of this job.

ediordna
  • 300
  • 2
  • 15

0 Answers0