0

My calculations using PyMC3 fail after pushing my operators to Google Cloud Composer. Errors provided below.

We've tried updating the PyPI packages of the compute environment and looked at the logs. This shows some clear errors, which seem to indicate that a compile job is failing on the cloud.

This is happens when airflow tries to parse our files - it is a fail on import.

Traceback (most recent call last): 

File "/opt/python3.6/lib/python3.6/site-packages/theano/gof/lazylinker_c.py", line 81, in <module> 
    actual_version, force_compile, _need_reload)) 

ImportError: Version check of the existing lazylinker compiled file. Looking for version 0.211, but found None. 

Extra debug information: force_compile=False, _need_reload=True
Traceback (most recent call last): 
File "/usr/local/lib/airflow/airflow/models.py", line 374, in 
    process_file m = imp.load_source(mod_name, filepath) 

File "/opt/python3.6/lib/python3.6/imp.py", line 172, in
    load_source module = _load(spec) 

File "<frozen importlib._bootstrap>", line 684, in 
    _load 

File "<frozen importlib._bootstrap>", line 665, in 
    _load_unlocked 

File "<frozen importlib._bootstrap_external>", line 678, in 
    exec_module 

File "<frozen importlib._bootstrap>", line 219, in 
    _call_with_frames_removed 

File "/home/airflow/gcs/dags/dag_abtesting/abtesting_orchestration.py", line 10, in <module> 
    from dag_abtesting.abtesting_tasks import task_analysis 

File "/home/airflow/gcs/dags/dag_abtesting/abtesting_tasks.py", line 16, in <module> 
    from dag_abtesting.task_bernoulli import task_analyse_bernoulli 

File "/home/airflow/gcs/dags/dag_abtesting/task_bernoulli.py", line 3, in <module> 
    from dag_abtesting.template_distributions import abstract_template_distribution, bernoulli_distribution 

File "/home/airflow/gcs/dags/dag_abtesting/template_distributions/bernoulli_distribution.py", line 6, in <module> 
    import pymc3 as pm 

File "/opt/python3.6/lib/python3.6/site-packages/pymc3/__init__.py", line 5, in <module> 
    from .distributions import * 

File "/opt/python3.6/lib/python3.6/site-packages/pymc3/distributions/__init__.py", line 1, in <module> 
    from . import timeseries 

File "/opt/python3.6/lib/python3.6/site-packages/pymc3/distributions/timeseries.py", line 1, in <module> 
    import theano.tensor as tt 

File "/opt/python3.6/lib/python3.6/site-packages/theano/__init__.py", line 110, in <module> 
    from theano.compile import ( 

File "/opt/python3.6/lib/python3.6/site-packages/theano/compile/__init__.py", line 12, in <module> 
    from theano.compile.mode import * 

File "/opt/python3.6/lib/python3.6/site-packages/theano/compile/mode.py", line 11, in <module> 
    import theano.gof.vm 

File "/opt/python3.6/lib/python3.6/site-packages/theano/gof/vm.py", line 674, in <module> 
    from . import lazylinker_c 

File "/opt/python3.6/lib/python3.6/site-packages/theano/gof/lazylinker_c.py", line 140, in <module> 
    preargs=args) 

File "/opt/python3.6/lib/python3.6/site-packages/theano/gof/cmodule.py", line 2396, in 
    compile_str (status, compile_stderr.replace('\n', '. '))) 

Exception: Compilation failed (return status=1): /usr/bin/ld: cannot find -lpython3.6m. collect2: error: ld returned 1 exit status.

It looks to me as if I'm missing something -likely a compiler - on my hosted environment? Can someone advise on a next action to take?

merv
  • 67,214
  • 13
  • 180
  • 245
Josko de Boer
  • 35
  • 2
  • 6
  • Perhaps related to [this question](https://stackoverflow.com/questions/50793797/importerror-version-check-of-the-existing-lazylinker-compiled-file-looking-for) or [this GitHub issue](https://github.com/bambinos/bambi/issues/160). Basically looks like your missing a library (`libpython`), which means either installing it or (if it's already installed) making sure it's on the linker path. – merv Sep 18 '19 at 22:36
  • That's interesting, @Merv. I'll check it out – Josko de Boer Sep 20 '19 at 07:44

0 Answers0