1

I have a Kedro pipeline that I want to run through a Python script, I think I have the minimum necessary code to do this, but everytime I try to run the pipeline through the script, I get a compatibility error regarding the protobuf version, but when I run the pipeline through the terminal it runs without problems. It is important to say that I am running everything inside a Docker container, and the image is based on PyTorch (version 1.9.0 and cuda 11.1).

This is the code I am using to call the pipeline:

from kedro.framework.context import load_context

class TBE():
  def run_inference():
    context = load_context('./')
    output = context.run(pipeline='inf')
    return output

And here is the error that I get when I run it:

[libprotobuf FATAL google/protobuf/stubs/common.cc:83] This program was compiled against 
version 3.9.2 of the Protocol Buffer runtime library, which is not compatible with the 
installed version (3.19.4).  Contact the program author for an update.  If you compiled 
the program yourself, make sure that your headers are from the same version of Protocol 
Buffers as your link-time library.  (Version verification failed in "bazel-out/k8- 
opt/bin/tensorflow/core/framework/tensor_shape.pb.cc".)
terminate called after throwing an instance of 'google::protobuf::FatalException'
what():  This program was compiled against version 3.9.2 of the Protocol Buffer runtime 
library, which is not compatible with the installed version (3.19.4).  Contact the 
program author for an update.  If you compiled the program yourself, make sure that your 
headers are from the same version of Protocol Buffers as your link-time library.  
(Version verification failed in "bazel-out/k8- 
opt/bin/tensorflow/core/framework/tensor_shape.pb.cc".)
Aborted

I have already tried changing the protobuf version, but I cannot find a compatible one. What can I do to solve this problem?

Gustavo Trivelatto
  • 11
  • 3

2 Answers2

1

I faced a similar problem with kedro. This helped:

pip install --upgrade "protobuf<=3.20.1"
BrokenBenchmark
  • 18,126
  • 7
  • 21
  • 33
Deep Contractor
  • 11
  • 1
0

Hello I'm not entirely sure if this will fix things - but the correct way to spin up a Kedro run like this is documented here and perhaps explains the discrepancy between what happens this way versus running kedro run in a terminal: https://kedro.readthedocs.io/en/stable/04_kedro_project_setup/03_session.html

datajoely
  • 1,466
  • 10
  • 13
  • Hello, thank you for the answer, I have tried the method shown in the documentation and still have the same problem. The only difference is that I didn't use the kedro metadata to get the package_name because my python couldn't find the module. Is there a chance that the metadata was doing something more than just passing the name? – Gustavo Trivelatto Mar 30 '22 at 19:05
  • So as I said - I wasn't sure that was going to fix things. Since the stack trace mentions incompatible protobuf versions - is there a chance you're not in the right venv? – datajoely Mar 31 '22 at 14:47